Note (2026-04-24): After this document was written, legal_entity was renamed to tenant and the old tenant was renamed to organization. Read references to these terms with the pre-rename meaning.

Document Edit History — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Persist inline edits of document.structured_data with a full audit trail, optimistic locking, and line-item CRUD/reorder. Replace the ingestion-to-document DB trigger with application-level transactional writes that go through the same audit trail.

Architecture: A new append-only document_history table stores RFC 6902 patches (extended with [id=X] for array-by-id paths) for both ingestion completions and user edits. document.structured_data stays the materialized current state; a new version column gives all-or-nothing optimistic locking. Four HTMX endpoints on /documents/:id/… return HTML fragments; client JS just calls htmx.ajax and lets HTMX swap. Provenance (which paths are human-edited) is computed on-the-fly from document_history at render time.

Tech Stack: Clojure 1.12, PostgreSQL 17 (JSONB, native uuidv7(), enums), Migratus, HoneySQL, next-jdbc, Reitit, Malli, Hiccup2, HTMX, SortableJS.

Spec: docs/superpowers/specs/2026-04-13-document-edit-history-design.md.

REPL workflow:

Verification: Most tasks have automated Clojure tests against embedded Postgres via the shared test/com/getorcha/test/fixtures.clj machinery. UI-layer tasks have manual verification steps via Playwright CLI (npx playwright-cli …) against the running dev server; no JS test framework in this repo.

File structure

New files:

Modified files:

Phase 1 — Foundation

Task 1: Add :id and :order to LineItem schemas

Files:

grep -n "^(def LineItem" src/com/getorcha/schema/*/structured_data.clj

Record which files need updating. Only update the ones that actually define a line-item schema.

Current (line 82-104):

(def LineItem
  "Individual invoice line item."
  [:map
   [:description :string]
   [:article-code [:maybe :string]]
   ;; … existing fields …
   [:page-location [:tuple :int :int]]])

Change to (new fields at the top of the map, immediately after the docstring):

(def LineItem
  "Individual invoice line item."
  [:map
   [:id :string]
   [:order :int]
   [:description :string]
   [:article-code [:maybe :string]]
   ;; … existing fields …
   [:page-location [:tuple :int :int]]])

Repeat for each other LineItem schema identified in Step 1.

cd orcha && clj -X:test:silent :nses '[com.getorcha.workers.ap.ingestion-test]' 2>&1 | grep -E "(FAIL|ERROR|Ran)"

Expected: existing tests that construct line-item fixtures without :id/:order will fail the schema validation. These will be fixed in Task 8; accept the breakage for now.

clj-kondo --lint src/com/getorcha/schema/invoice/structured_data.clj

Expected: no errors or warnings.

git add src/com/getorcha/schema
git commit -m "schema: require :id and :order on LineItem"

Task 2: Add :document/version to the Document Malli schema

Files:

Current:

(def Document
  "…"
  (m/schema
   [:map
    [:document/id ID]
    [:document/legal-entity-id schema.legal-entity/ID]
    ;; … existing fields …
    [:document/created-at inst?]
    [:document/updated-at inst?]]))

Insert [:document/version :int] immediately after [:document/id ID]:

(def Document
  "…"
  (m/schema
   [:map
    [:document/id ID]
    [:document/version :int]
    [:document/legal-entity-id schema.legal-entity/ID]
    ;; … existing fields …
    [:document/created-at inst?]
    [:document/updated-at inst?]]))

clj-kondo --lint src/com/getorcha/schema/document.clj

git add src/com/getorcha/schema/document.clj
git commit -m "schema: add :document/version to Document"

Task 3: Create PENDING-CLEANUPS.md

Files:

# Pending schema cleanups

Tracks columns, tables, and triggers that are no longer written or read
but have not yet been dropped. Each entry states what's stale, what
replaces it, and the gating condition for removal.

## `ingestion.structured_data` (JSONB)

- **Replaced by:** `document_history.patch` (for per-ingestion state)
  and `document.structured_data` (for the current materialized state).
- **Stopped being written:** the migration that added
  `document_history` and dropped `trg_update_document_from_ingestion`.
  After that point the ingestion worker's completion handler writes
  to `document_history` + `document` transactionally.
- **Gate to drop:** the migration above has been verified stable in
  production with no open regressions referencing this column.

## `ingestion.valid_structured_data` (BOOLEAN)

- **Replaced by:** Malli schema validation in the ingestion worker,
  recorded implicitly by the presence of a `document_history` row
  with `change_type='ingestion'` (failed validation means no row).
- **Stopped being written:** same as above.
- **Gate to drop:** same as above.

git add resources/migrations/PENDING-CLEANUPS.md
git commit -m "docs: track deferred drop of ingestion.structured_data columns"

Task 4: Write the migration DDL (table + enum + version column)

Files:

Migratus uses sortable filename-prefix timestamps, e.g. 20260413120000-…. Use date +%Y%m%d%H%M%S for the timestamp.

-- up

CREATE TYPE document_history_change_type AS ENUM ('ingestion', 'edit');

CREATE TABLE document_history (
    id            UUID        PRIMARY KEY DEFAULT uuidv7(),
    document_id   UUID        NOT NULL REFERENCES document(id) ON DELETE CASCADE,
    change_type   document_history_change_type NOT NULL,
    ingestion_id  UUID        REFERENCES ingestion(id) ON DELETE SET NULL,
    edited_by     UUID        REFERENCES "identity"(id) ON DELETE SET NULL,
    patch         JSONB       NOT NULL,
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),

    CONSTRAINT document_history_source_xor CHECK (
        (change_type = 'ingestion' AND ingestion_id IS NOT NULL AND edited_by IS NULL)
        OR
        (change_type = 'edit'      AND edited_by    IS NOT NULL AND ingestion_id IS NULL)
    )
);

CREATE INDEX idx_document_history_document_created
    ON document_history(document_id, created_at);

ALTER TABLE document ADD COLUMN version INT NOT NULL DEFAULT 1;

-- down

ALTER TABLE document DROP COLUMN IF EXISTS version;
DROP INDEX IF EXISTS idx_document_history_document_created;
DROP TABLE IF EXISTS document_history;
DROP TYPE IF EXISTS document_history_change_type;

psql -h localhost -U postgres -d orcha -c "\d document_history" 2>&1 || true
cd orcha && clj -X:dev com.getorcha.db.migrations/migrate
psql -h localhost -U postgres -d orcha -c "\d document_history"
psql -h localhost -U postgres -d orcha -c "\d document" | grep version

Expected: table exists with all columns; document has a version column, default 1.

cd orcha && clj -X:dev com.getorcha.db.migrations/rollback
psql -h localhost -U postgres -d orcha -c "\d document_history" 2>&1 | grep -q "Did not find" && echo "rolled back OK"
cd orcha && clj -X:dev com.getorcha.db.migrations/migrate

git add resources/migrations/*-add-document-history.*
git commit -m "migration: add document_history table and document.version"

Task 5: Extend the migration to backfill + drop the trigger

Files:

After the DDL, add:

-- Backfill :id and :order onto existing line items.
-- idx is 1-based from WITH ORDINALITY; we want 0-based, hence (idx - 1).
UPDATE document SET structured_data = (
    SELECT jsonb_set(
        structured_data,
        '{line-items}',
        (SELECT jsonb_agg(
            item || jsonb_build_object(
                'id',    gen_random_uuid()::text,
                'order', (idx - 1)
            ) ORDER BY idx
        )
        FROM jsonb_array_elements(structured_data->'line-items')
             WITH ORDINALITY AS t(item, idx))
    )
)
WHERE structured_data ? 'line-items'
  AND jsonb_typeof(structured_data->'line-items') = 'array';

-- Backfill one history row per document from its most recent successful ingestion.
INSERT INTO document_history (document_id, change_type, ingestion_id, patch, created_at)
SELECT
    d.id,
    'ingestion'::document_history_change_type,
    latest.id,
    jsonb_build_array(jsonb_build_object(
        'op',    'replace',
        'path',  '',
        'value', d.structured_data
    )),
    COALESCE(latest.completed_at, d.created_at)
FROM document d
LEFT JOIN LATERAL (
    SELECT id, completed_at
    FROM ingestion
    WHERE ingestion.document_id = d.id
      AND status = 'completed'
    ORDER BY completed_at DESC
    LIMIT 1
) latest ON TRUE
WHERE d.structured_data IS NOT NULL;

-- Drop the trigger; app-level code now handles ingestion → document.
DROP TRIGGER IF EXISTS trg_update_document_from_ingestion ON ingestion;
DROP FUNCTION IF EXISTS update_document_from_ingestion();

At the END of the down file (before the final table drops), add the trigger function back verbatim from resources/migrations/init.sql:276-304:

-- Recreate the dropped trigger
CREATE OR REPLACE FUNCTION update_document_from_ingestion()
RETURNS TRIGGER AS $$
BEGIN
    IF NEW.status = 'completed' AND OLD.status = 'in-progress' THEN
        UPDATE document
        SET
            type = 'invoice',
            structured_data = NEW.structured_data,
            needs_human_review = (
                NOT COALESCE(NEW.valid_structured_data, true)
                OR EXISTS (
                    SELECT 1 FROM jsonb_each(NEW.structured_data->'validation-results') AS v(k, val)
                    WHERE val->>'status' = 'error'
                )
            ),
            updated_at = now()
        WHERE id = NEW.document_id;
    END IF;
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trg_update_document_from_ingestion
    AFTER UPDATE ON ingestion
    FOR EACH ROW
    EXECUTE FUNCTION update_document_from_ingestion();

Note: the trigger references NEW.structured_data — it will only work after a rollback if ingestion.structured_data still contains data (which it will for pre-migration rows). For new ingestions made against the new code and then rolled back, the column will be NULL and the trigger is a no-op.

First seed a document with line items (or use any existing one). Then:

cd orcha && clj -X:dev com.getorcha.db.migrations/rollback
cd orcha && clj -X:dev com.getorcha.db.migrations/migrate
psql -h localhost -U postgres -d orcha -c "SELECT id, jsonb_path_query_array(structured_data, '\$.\"line-items\"[*].id') AS item_ids FROM document WHERE structured_data ? 'line-items' LIMIT 3" -x
psql -h localhost -U postgres -d orcha -c "SELECT document_id, change_type, jsonb_array_length(patch) AS ops FROM document_history LIMIT 5" -x

Expected: line-items have UUID ids; each document with structured_data has at least one history row of change_type='ingestion' with 1-op patch.

git add resources/migrations/*-add-document-history.*
git commit -m "migration: backfill document_history and drop the ingestion-to-document trigger"

Task 6: JSON Patch path helper — Clojure vector ↔ JSON Pointer

Files:

Design: callers in view code specify paths as Clojure vectors. Segments are keywords (object keys), ints (array indices), or maps of the form {:id "<uuid>"} (array-by-id lookup — rendered as [id=X] appended to the preceding segment). A leading :structured-data is stripped for convenience.

(ns com.getorcha.json-patch.path-test
  (:require [clojure.test :refer [deftest is testing]]
            [com.getorcha.json-patch.path :as path]))


(deftest clj-path->pointer-test
  (testing "empty path (root)"
    (is (= "" (path/clj-path->pointer []))))

  (testing "scalar key"
    (is (= "/invoice-number"
           (path/clj-path->pointer [:invoice-number]))))

  (testing "nested scalar"
    (is (= "/issuer/name"
           (path/clj-path->pointer [:issuer :name]))))

  (testing "leading :structured-data is stripped"
    (is (= "/invoice-number"
           (path/clj-path->pointer [:structured-data :invoice-number]))))

  (testing "numeric array index"
    (is (= "/line-items/0/amount"
           (path/clj-path->pointer [:line-items 0 :amount]))))

  (testing "array-by-id segment attaches to preceding key"
    (is (= "/line-items[id=li-abc]/amount"
           (path/clj-path->pointer [:line-items {:id "li-abc"} :amount]))))

  (testing "array-by-id at the tail"
    (is (= "/line-items[id=li-abc]"
           (path/clj-path->pointer [:line-items {:id "li-abc"}]))))

  (testing "deeply nested with :structured-data prefix and id lookup"
    (is (= "/line-items[id=li-abc]/debit-account/number"
           (path/clj-path->pointer [:structured-data :line-items {:id "li-abc"}
                                    :debit-account :number])))))


(deftest pointer->clj-path-test
  (testing "empty (root)"
    (is (= [] (path/pointer->clj-path ""))))

  (testing "scalar"
    (is (= [:invoice-number] (path/pointer->clj-path "/invoice-number"))))

  (testing "nested scalar"
    (is (= [:issuer :name] (path/pointer->clj-path "/issuer/name"))))

  (testing "array-by-id segment"
    (is (= [:line-items {:id "li-abc"} :amount]
           (path/pointer->clj-path "/line-items[id=li-abc]/amount"))))

  (testing "numeric array index"
    (is (= [:line-items 0 :amount]
           (path/pointer->clj-path "/line-items/0/amount")))))

cd orcha && clj -X:test:silent :nses '[com.getorcha.json-patch.path-test]' 2>&1 | grep -E "(FAIL|ERROR|Ran)"

(ns com.getorcha.json-patch.path
  "Bidirectional conversion between Clojure vector paths and JSON Pointer
   strings extended with `[id=X]` array-by-id segments.

   Vector form: `[:structured-data :line-items {:id \"li-abc\"} :amount]`
   Pointer:     `/line-items[id=li-abc]/amount`

   Leading `:structured-data` in the vector form is stripped — the patches
   operate on the `structured-data` value directly."
  (:require [clojure.string :as string]))


(defn ^:private segment->pointer-piece
  [seg]
  (cond
    (keyword? seg) (name seg)
    (int? seg)     (str seg)
    :else (throw (ex-info "unsupported path segment" {:segment seg}))))


(defn clj-path->pointer
  "Returns the JSON Pointer string for `path`. Returns `\"\"` for root."
  [path]
  (let [segs (if (= :structured-data (first path)) (rest path) path)]
    (if (empty? segs)
      ""
      (loop [[seg & more] segs
             out          ""]
        (cond
          (nil? seg)
          out

          (and (map? seg) (contains? seg :id))
          (recur more (str out "[id=" (:id seg) "]"))

          :else
          (recur more (str out "/" (segment->pointer-piece seg))))))))


(defn ^:private parse-pointer-segment
  "Parses a single segment like `line-items[id=li-abc]` into
   `[:line-items {:id \"li-abc\"}]` or simply `[:plain-key]`."
  [piece]
  (if-let [[_ base id] (re-matches #"([^\[]+)\[id=([^\]]+)\]" piece)]
    [(keyword base) {:id id}]
    [(if (re-matches #"\d+" piece)
       (parse-long piece)
       (keyword piece))]))


(defn pointer->clj-path
  "Parses a JSON Pointer `pointer` (with optional `[id=X]` segments)
   into a Clojure vector path."
  [pointer]
  (if (= "" pointer)
    []
    (into [] (mapcat parse-pointer-segment)
          (rest (string/split pointer #"/")))))

cd orcha && clj -X:test:silent :nses '[com.getorcha.json-patch.path-test]' 2>&1 | grep -E "(FAIL|ERROR|Ran)"

Expected: Ran N tests … 0 failures, 0 errors.

clj-kondo --lint src/com/getorcha/json_patch test/com/getorcha/json_patch

git add src/com/getorcha/json_patch test/com/getorcha/json_patch
git commit -m "feat: json-patch path helper"

Task 7: JSON Patch applier (RFC 6902 with [id=X] extension)

Files:

Design: one entry point, apply-patch, that takes a document value and a patch (vector of ops, each a map with "op", "path", and "value"/"from" as needed) and returns the new value — or throws a typed exception on invalid input.

Supported ops: "replace", "add", "remove". No other ops for MVP (no "move", "copy", "test"). Apply rules per RFC 6902, except array lookup by [id=X] rather than numeric index when the path segment is in that form.

(ns com.getorcha.json-patch-test
  (:require [clojure.test :refer [deftest is testing]]
            [com.getorcha.json-patch :as jp]))


(deftest replace-scalar
  (testing "at root"
    (is (= {:a 2}
           (jp/apply-patch {:a 1}
                           [{"op" "replace" "path" "" "value" {:a 2}}]))))

  (testing "nested scalar"
    (is (= {:invoice-number "INV-99"}
           (jp/apply-patch {:invoice-number "INV-0"}
                           [{"op" "replace" "path" "/invoice-number"
                             "value" "INV-99"}]))))

  (testing "missing path raises :path-not-found"
    (is (thrown-with-msg? clojure.lang.ExceptionInfo
                          #"path-not-found"
                          (jp/apply-patch {}
                                          [{"op" "replace"
                                            "path" "/nope"
                                            "value" 1}])))))


(deftest replace-array-by-id
  (let [doc {:line-items [{:id "a" :amount 100}
                          {:id "b" :amount 200}]}]
    (testing "replace a sub-field by id"
      (is (= {:line-items [{:id "a" :amount 100}
                           {:id "b" :amount 999}]}
             (jp/apply-patch doc
                             [{"op" "replace"
                               "path" "/line-items[id=b]/amount"
                               "value" 999}]))))
    (testing "id not resolvable raises :id-not-resolvable"
      (is (thrown-with-msg? clojure.lang.ExceptionInfo
                            #"id-not-resolvable"
                            (jp/apply-patch doc
                                            [{"op" "replace"
                                              "path" "/line-items[id=zz]/amount"
                                              "value" 1}]))))))


(deftest add-to-array
  (testing "append via `-`"
    (is (= {:line-items [{:id "a"} {:id "b"}]}
           (jp/apply-patch {:line-items [{:id "a"}]}
                           [{"op" "add"
                             "path" "/line-items/-"
                             "value" {:id "b"}}])))))


(deftest remove-by-id
  (is (= {:line-items [{:id "a"}]}
         (jp/apply-patch {:line-items [{:id "a"} {:id "b"}]}
                         [{"op" "remove"
                           "path" "/line-items[id=b]"}]))))


(deftest multi-op-patch
  (testing "reorder via multiple replaces"
    (is (= {:line-items [{:id "a" :order 1}
                         {:id "b" :order 0}]}
           (jp/apply-patch {:line-items [{:id "a" :order 0}
                                         {:id "b" :order 1}]}
                           [{"op" "replace"
                             "path" "/line-items[id=a]/order"
                             "value" 1}
                            {"op" "replace"
                             "path" "/line-items[id=b]/order"
                             "value" 0}])))))


(deftest root-replace-for-ingestion
  (let [new-value {:invoice-number "INV-1" :line-items [{:id "x"}]}]
    (is (= new-value
           (jp/apply-patch {:invoice-number "old"}
                           [{"op" "replace" "path" "" "value" new-value}])))))

cd orcha && clj -X:test:silent :nses '[com.getorcha.json-patch-test]' 2>&1 | grep -E "(FAIL|ERROR|Ran)"

(ns com.getorcha.json-patch
  "In-house applier for RFC 6902 JSON Patch with one extension: array
   elements may be addressed by `[id=X]` instead of numeric index.

   Supported ops: `replace`, `add`, `remove`. Throws `ex-info` with
   `:type` of `:path-not-found`, `:id-not-resolvable`, `:unsupported-op`,
   or `:bad-patch` on failures."
  (:require [clojure.string :as string]))


(defn ^:private resolve-segment
  "Given a value and a parsed segment, returns the next value or throws."
  [val seg]
  (cond
    (and (map? seg) (contains? seg :id))
    (if-let [idx (some (fn [[i m]] (when (= (:id seg) (get m "id")) i))
                       (map-indexed vector val))]
      [idx (nth val idx)]
      (throw (ex-info "id-not-resolvable"
                      {:type :id-not-resolvable :segment seg})))

    (keyword? seg)
    (let [k (name seg)]
      (if (contains? val k)
        [k (get val k)]
        (throw (ex-info "path-not-found"
                        {:type :path-not-found :segment seg}))))

    (int? seg)
    (if (<= 0 seg (dec (count val)))
      [seg (nth val seg)]
      (throw (ex-info "path-not-found"
                      {:type :path-not-found :segment seg})))

    :else
    (throw (ex-info "bad-patch"
                    {:type :bad-patch :segment seg}))))


(defn ^:private parse-segments
  "Parses a JSON Pointer to a vector of segment parsers usable by
   `resolve-segment`. Uses the same logic as the path helper but is
   inlined here to avoid a circular require."
  [pointer]
  (if (= "" pointer)
    []
    (into []
          (mapcat (fn [piece]
                    (if-let [[_ base id] (re-matches #"([^\[]+)\[id=([^\]]+)\]" piece)]
                      [(keyword base) {:id id}]
                      [(if (re-matches #"\d+" piece)
                         (parse-long piece)
                         (keyword piece))])))
          (rest (string/split pointer #"/")))))


(defn ^:private assoc-at
  "Recursively replaces the value at `segs` inside `val` with `new-val`."
  [val segs new-val]
  (if (empty? segs)
    new-val
    (let [[seg & more] segs
          [k _] (resolve-segment val seg)]
      (cond
        (vector? val) (assoc val k (assoc-at (nth val k) more new-val))
        (map? val)    (assoc val k (assoc-at (get val k) more new-val))))))


(defn ^:private dissoc-at
  "Removes the element/key at `segs` inside `val`."
  [val segs]
  (let [[seg & more] segs
        [k _] (resolve-segment val seg)]
    (cond
      (empty? more)
      (cond
        (vector? val) (into (subvec val 0 k) (subvec val (inc k)))
        (map? val)    (dissoc val k))

      (vector? val)
      (assoc val k (dissoc-at (nth val k) more))

      (map? val)
      (assoc val k (dissoc-at (get val k) more)))))


(defn ^:private apply-add
  [val pointer added]
  (cond
    ;; "/line-items/-" appends to the array at that parent.
    (string/ends-with? pointer "/-")
    (let [parent-pointer (subs pointer 0 (- (count pointer) 2))
          parent-segs    (parse-segments parent-pointer)]
      (assoc-at val parent-segs
                (conj (get-in val (mapv (fn [s]
                                          (if (keyword? s) (name s) s))
                                        parent-segs))
                      added)))

    :else
    (assoc-at val (parse-segments pointer) added)))


(defn ^:private apply-op
  [val {:strs [op path value] :as operation}]
  (case op
    "replace"
    (let [segs (parse-segments path)]
      (if (empty? segs)
        value
        (assoc-at val segs value)))

    "add"
    (apply-add val path value)

    "remove"
    (dissoc-at val (parse-segments path))

    (throw (ex-info "unsupported-op"
                    {:type :unsupported-op :op op :operation operation}))))


(defn apply-patch
  "Applies a vector of RFC 6902-ish ops to `document-value` and returns
   the new value. Ops are applied in order; the result of one op is the
   input to the next. Throws on any failure — callers should catch
   `ExceptionInfo` with `(:type (ex-data e))`."
  [document-value patch]
  (reduce apply-op document-value patch))

cd orcha && clj -X:test:silent :nses '[com.getorcha.json-patch-test]' 2>&1 | grep -E "(FAIL|ERROR|Ran)"

clj-kondo --lint src/com/getorcha/json_patch.clj test/com/getorcha/json_patch_test.clj

git add src/com/getorcha/json_patch.clj test/com/getorcha/json_patch_test.clj
git commit -m "feat: json-patch applier with [id=X] array extension"

Task 8: document_history schema + DB helpers

Files:

(ns com.getorcha.schema.document-history
  "Schema for document_history rows — audit log of ingestion completions
   and user edits on document.structured_data."
  (:require [com.getorcha.schema.document :as schema.document]
            [com.getorcha.schema.ingestion :as schema.ingestion]
            [malli.core :as m]))


(def ChangeType [:enum :ingestion :edit])


(def DocumentHistory
  (m/schema
   [:map
    [:document-history/id :uuid]
    [:document-history/document-id schema.document/ID]
    [:document-history/change-type ChangeType]
    [:document-history/ingestion-id [:maybe schema.ingestion/ID]]
    [:document-history/edited-by [:maybe :uuid]]
    [:document-history/patch [:vector :map]]
    [:document-history/created-at inst?]]))

(ns com.getorcha.db.document-history-test
  (:require [clojure.test :refer [deftest is testing use-fixtures]]
            [com.getorcha.db :as db]
            [com.getorcha.db.document-history :as dh]
            [com.getorcha.db.sql :as db.sql]
            [com.getorcha.test.fixtures :as fixtures]))


(use-fixtures :once fixtures/system-fixture)
(use-fixtures :each fixtures/db-transaction-fixture)


(defn ^:private seed-document!
  []
  (let [le-id (db.sql/execute-one!
               fixtures/*db*
               {:select [:id] :from [:legal-entity] :limit 1})
        id    (random-uuid)]
    (db.sql/execute-one!
     fixtures/*db*
     {:insert-into :document
      :values [{:id id
                :legal-entity-id (:legal-entity/id le-id)
                :content-hash (str "test-" id)
                :file-path "x.pdf"
                :structured-data [:cast (cheshire.core/generate-string
                                         {:document-type "invoice"
                                          :invoice-number "INV-1"})
                                  :jsonb]}]})
    id))


(deftest insert-and-fetch-history
  (testing "inserting an edit row with patch, then fetching newest→oldest"
    (let [doc-id (seed-document!)
          identity-id (:identity/id (db.sql/execute-one!
                                     fixtures/*db*
                                     {:select [:id] :from [:identity] :limit 1}))]
      (dh/insert! fixtures/*db*
                  {:document-id doc-id
                   :change-type :edit
                   :edited-by   identity-id
                   :patch       [{"op" "replace"
                                  "path" "/invoice-number"
                                  "value" "INV-2"}]})
      (let [rows (dh/rows-for-document fixtures/*db* doc-id)]
        (is (= 1 (count rows)))
        (is (= :edit (:document-history/change-type (first rows))))
        (is (= [{"op" "replace" "path" "/invoice-number" "value" "INV-2"}]
               (:document-history/patch (first rows))))))))

cd orcha && clj -X:test:silent :nses '[com.getorcha.db.document-history-test]' 2>&1 | grep -E "(FAIL|ERROR|Ran)"

(ns com.getorcha.db.document-history
  "Query and insert helpers for document_history."
  (:require [cheshire.core :as json]
            [com.getorcha.db :as db]
            [com.getorcha.db.sql :as db.sql]
            [com.getorcha.schema.document-history :as schema.document-history]))


(def ^:private row-builder
  (db/schema-row-builder schema.document-history/DocumentHistory))


(defn insert!
  "Inserts a history row. `row` keys: :document-id, :change-type
   (:ingestion or :edit), :ingestion-id or :edited-by (exactly one),
   :patch (vector of op maps)."
  [db-pool {:keys [document-id change-type ingestion-id edited-by patch]}]
  (db.sql/execute-one!
   db-pool
   {:insert-into :document-history
    :values [{:document-id  document-id
              :change-type  [:cast (name change-type)
                             :document-history-change-type]
              :ingestion-id ingestion-id
              :edited-by    edited-by
              :patch        [:cast (json/generate-string patch) :jsonb]}]}))


(defn rows-for-document
  "Returns all history rows for a document, ordered newest first."
  [db-pool document-id]
  (db.sql/execute!
   db-pool
   {:select   [:*]
    :from     [:document-history]
    :where    [:= :document-id document-id]
    :order-by [[:created-at :desc]]}
   {:builder-fn row-builder}))

cd orcha && clj -X:test:silent :nses '[com.getorcha.db.document-history-test]' 2>&1 | grep -E "(FAIL|ERROR|Ran)"

clj-kondo --lint src/com/getorcha/db/document_history.clj src/com/getorcha/schema/document_history.clj test/com/getorcha/db/document_history_test.clj

git add src/com/getorcha/schema/document_history.clj src/com/getorcha/db/document_history.clj test/com/getorcha/db/document_history_test.clj
git commit -m "feat: document_history schema and db helpers"

Phase 2 — Ingestion integration

Task 9: Annotate line-items with :id/:order during ingestion

Files:

Search for the final validate call (where schema validation runs, around the current line ~340 per the earlier grep):

grep -n "validate\|structured-data" src/com/getorcha/workers/ap/ingestion.clj | head -20

Find the function that runs validation (the function body including (update ingestion :structured-data validation/validate legal-entity)). The annotation must happen before this call.

Add a private function near the top of the file (after the ns form):

(defn ^:private annotate-line-items
  "Adds :id (random UUID string) and :order (0-based index) to every
   line item in structured-data. Idempotent: existing ids and orders
   are preserved."
  [structured-data]
  (if (seq (:line-items structured-data))
    (update structured-data :line-items
            (fn [items]
              (vec (map-indexed
                    (fn [idx item]
                      (-> item
                          (update :id    #(or % (str (random-uuid))))
                          (update :order #(or % idx))))
                    items))))
    structured-data))

Locate the validation step in the pipeline. Replace

(update ingestion :structured-data validation/validate legal-entity)

with

(-> ingestion
    (update :structured-data annotate-line-items)
    (update :structured-data validation/validate legal-entity))

If the actual structure differs, insert the annotate-line-items call directly before whatever invokes validation, threading through the same structured-data value.

Append to test/com/getorcha/workers/ap/ingestion_test.clj:

(deftest annotate-line-items-assigns-id-and-order
  (testing "line-items without id or order get both"
    (let [out (#'com.getorcha.workers.ap.ingestion/annotate-line-items
               {:line-items [{:description "a"}
                             {:description "b"}]})]
      (is (every? (fn [{:keys [id order]}]
                    (and (string? id) (int? order)))
                  (:line-items out)))
      (is (= [0 1] (mapv :order (:line-items out))))))

  (testing "pre-existing ids are preserved"
    (let [out (#'com.getorcha.workers.ap.ingestion/annotate-line-items
               {:line-items [{:id "already-set" :description "a"}]})]
      (is (= "already-set" (-> out :line-items first :id))))))

cd orcha && clj -X:test:silent :nses '[com.getorcha.workers.ap.ingestion-test]' 2>&1 | grep -A 3 -E "(FAIL|ERROR|Ran)"

Expected: the two new tests pass. Pre-existing ingestion tests that construct fixtures without :id/:order may still fail at the schema step — those are fixed in Task 10.

clj-kondo --lint src/com/getorcha/workers/ap/ingestion.clj test/com/getorcha/workers/ap/ingestion_test.clj

git add src/com/getorcha/workers/ap/ingestion.clj test/com/getorcha/workers/ap/ingestion_test.clj
git commit -m "feat(ingestion): annotate line-items with :id and :order"

Task 10: Replace the trigger with a transactional completion handler

Files:

grep -n "update.*ingestion.*status\|:status.*completed\|next-jdbc\|with-transaction" src/com/getorcha/workers/ap/ingestion.clj | head

Find the function that finalizes an ingestion (where it sets status = :completed). In the current code this is a single UPDATE ingestion that relies on the trigger to copy structured_data into document.

Replace the existing finalize block with:

(defn ^:private derive-needs-human-review?
  [structured-data valid-structured-data?]
  (or (not (boolean valid-structured-data?))
      (some (fn [[_ v]] (= "error" (get v :status)))
            (:validation-results structured-data))))


(defn ^:private finalize-ingestion!
  "Commits a successful ingestion: writes document_history row, updates
   document.structured_data + version + type + needs_human_review, and
   marks the ingestion row completed — all in one transaction."
  [db-pool {:ap-ingestion/keys [id document-id structured-data valid-structured-data] :as _ingestion}]
  (jdbc/with-transaction [tx db-pool]
    (jdbc.sql/execute! tx {:select [:version]
                           :from   [:document]
                           :where  [:= :id document-id]
                           :for    :update})
    (com.getorcha.db.document-history/insert!
     tx
     {:document-id document-id
      :change-type :ingestion
      :ingestion-id id
      :patch [{"op" "replace"
               "path" ""
               "value" structured-data}]})
    (db.sql/execute-one!
     tx
     {:update :document
      :set    {:structured-data    [:cast (cheshire.core/generate-string structured-data)
                                    :jsonb]
               :type               [:cast (name (keyword (:document-type structured-data)))
                                    :document-type]
               :needs-human-review (derive-needs-human-review?
                                    structured-data valid-structured-data)
               :version            [:raw "version + 1"]
               :updated-at         (java.time.Instant/now)}
      :where  [:= :id document-id]})
    (db.sql/execute-one!
     tx
     {:update :ingestion
      :set    {:status       [:cast "completed" :ingestion-status]
               :completed-at (java.time.Instant/now)}
      :where  [:= :id id]})))

Add the requires needed at the top (alphabetized):

[cheshire.core]
[com.getorcha.db.document-history]
[next.jdbc :as jdbc]

Call finalize-ingestion! at the end of a successful pipeline in place of the previous single-UPDATE. Ensure structured-data and valid-structured-data are the already-annotated, validated values.

In test/com/getorcha/workers/ap/ingestion_test.clj, wherever tests build a :line-items vector for fixtures, add :id (str (random-uuid)) :order <idx> to each item. Also, where tests previously asserted against :ap-ingestion/structured-data, switch to:

(let [doc (db.sql/execute-one!
           fixtures/*db*
           {:select [:*] :from [:document] :where [:= :id doc-id]}
           {:builder-fn shared/document-builder-fn})]
  (is (= "INV-99" (-> doc :document/structured-data :invoice-number)))
  (is (= 2 (:document/version doc))))

(let [history (com.getorcha.db.document-history/rows-for-document fixtures/*db* doc-id)]
  (is (= 1 (count history)))
  (is (= :ingestion (-> history first :document-history/change-type))))

cd orcha && clj -X:test:silent :nses '[com.getorcha.workers.ap.ingestion-test]' 2>&1 | grep -A 5 -E "(FAIL|ERROR|Ran)"

Expected: all green. If specific tests still reference ap-ingestion/structured-data, migrate each.

clj-kondo --lint src/com/getorcha/workers/ap/ingestion.clj test/com/getorcha/workers/ap/ingestion_test.clj

git add src/com/getorcha/workers/ap/ingestion.clj test/com/getorcha/workers/ap/ingestion_test.clj
git commit -m "feat(ingestion): transactional completion writes document_history"

Task 11: Remove structured-data / valid-structured-data from the ingestion Malli schema

Files:

Delete these lines from the Ingestion schema:

    [:ap-ingestion/structured-data [:maybe schema.structured-data/StructuredData]]
    [:ap-ingestion/valid-structured-data [:maybe :boolean]]

Also remove the require for schema.structured-data if it becomes unused.

clj-kondo --lint src/com/getorcha/schema/ingestion.clj
cd orcha && clj -X:test:silent 2>&1 | grep -E "(FAIL|ERROR|Ran .* tests)"

Expected: no new failures. Any existing test that reads :ap-ingestion/structured-data should be updated to read :document/structured-data instead (see Task 10 Step 3 for the pattern).

git add src/com/getorcha/schema/ingestion.clj
git commit -m "schema(ingestion): remove structured-data fields (column kept per PENDING-CLEANUPS)"

Phase 3 — Read path

Task 12: document-provenance helper

Files:

(ns com.getorcha.app.http.documents.view.provenance-test
  (:require [clojure.test :refer [deftest is testing use-fixtures]]
            [com.getorcha.app.http.documents.view.provenance :as provenance]
            [com.getorcha.db.document-history :as dh]
            [com.getorcha.db.sql :as db.sql]
            [com.getorcha.test.fixtures :as fixtures]))


(use-fixtures :once fixtures/system-fixture)
(use-fixtures :each fixtures/db-transaction-fixture)


(defn ^:private seed-doc-with-ingestion!
  "Inserts a document, an ingestion row, and a history 'ingestion' row.
   Returns {:doc-id, :ingestion-id, :identity-id}."
  [structured-data]
  (let [le-id       (:legal-entity/id
                     (db.sql/execute-one! fixtures/*db*
                                          {:select [:id] :from [:legal-entity] :limit 1}))
        identity-id (:identity/id
                     (db.sql/execute-one! fixtures/*db*
                                          {:select [:id] :from [:identity] :limit 1}))
        doc-id      (random-uuid)
        ingestion-id (random-uuid)]
    (db.sql/execute-one!
     fixtures/*db*
     {:insert-into :document
      :values [{:id              doc-id
                :legal-entity-id le-id
                :content-hash    (str "hash-" doc-id)
                :file-path       "x.pdf"
                :version         1
                :structured-data [:cast (cheshire.core/generate-string structured-data)
                                  :jsonb]}]})
    (db.sql/execute-one!
     fixtures/*db*
     {:insert-into :ingestion
      :values [{:id          ingestion-id
                :document-id doc-id
                :uploaded-by identity-id
                :status      [:cast "completed" :ingestion-status]}]})
    (dh/insert! fixtures/*db*
                {:document-id  doc-id
                 :change-type  :ingestion
                 :ingestion-id ingestion-id
                 :patch        [{"op" "replace" "path" "" "value" structured-data}]})
    {:doc-id doc-id :ingestion-id ingestion-id :identity-id identity-id}))


(deftest empty-when-no-edits
  (let [{:keys [doc-id]} (seed-doc-with-ingestion!
                          {:invoice-number "INV-1"
                           :line-items []})]
    (is (= {} (provenance/document-provenance fixtures/*db* doc-id)))))


(deftest single-edit-returns-map
  (let [{:keys [doc-id identity-id]} (seed-doc-with-ingestion!
                                      {:invoice-number "INV-1"})]
    (dh/insert! fixtures/*db*
                {:document-id doc-id
                 :change-type :edit
                 :edited-by identity-id
                 :patch [{"op" "replace"
                          "path" "/invoice-number"
                          "value" "INV-2"}]})
    (let [pm (provenance/document-provenance fixtures/*db* doc-id)]
      (is (= 1 (count pm)))
      (is (= identity-id (-> pm (get "/invoice-number") :edited-by))))))


(deftest newest-edit-at-path-wins
  (let [{:keys [doc-id identity-id]} (seed-doc-with-ingestion!
                                      {:invoice-number "INV-1"})]
    (dh/insert! fixtures/*db*
                {:document-id doc-id :change-type :edit :edited-by identity-id
                 :patch [{"op" "replace" "path" "/invoice-number" "value" "INV-2"}]})
    (Thread/sleep 10)
    (dh/insert! fixtures/*db*
                {:document-id doc-id :change-type :edit :edited-by identity-id
                 :patch [{"op" "replace" "path" "/invoice-number" "value" "INV-3"}]})
    (let [{:keys [edited-at]} (get (provenance/document-provenance fixtures/*db* doc-id)
                                   "/invoice-number")]
      (is (some? edited-at)))))


(deftest walker-stops-at-most-recent-ingestion
  (let [{:keys [doc-id ingestion-id identity-id]}
        (seed-doc-with-ingestion! {:invoice-number "INV-1"})]
    ;; Pre-ingestion edit (older) — should be ignored.
    (dh/insert! fixtures/*db*
                {:document-id doc-id :change-type :edit :edited-by identity-id
                 :patch [{"op" "replace" "path" "/invoice-number" "value" "IGNORED"}]})
    ;; Then an ingestion row (fresh extraction) — supersedes the edit above.
    (dh/insert! fixtures/*db*
                {:document-id doc-id :change-type :ingestion :ingestion-id ingestion-id
                 :patch [{"op" "replace" "path" "" "value" {:invoice-number "INV-2"}}]})
    (is (= {} (provenance/document-provenance fixtures/*db* doc-id)))))

Implement the seed-doc-with-ingestion! helper (inline at Step 2 of this task) using direct SQL inserts into document, ingestion, identity, and document_history.

(ns com.getorcha.app.http.documents.view.provenance
  "Computes a path → {:edited-by, :edited-at} map for a document's
   structured-data, by walking document_history rows since the most
   recent change_type='ingestion'."
  (:require [com.getorcha.db.document-history :as dh]))


(defn document-provenance
  "Returns {json-pointer-path-string → {:edited-by, :edited-at}} for every
   path with a human edit since the most recent ingestion for this document.
   Paths absent from the map are implicitly LLM-sourced."
  [db-pool document-id]
  (let [rows        (dh/rows-for-document db-pool document-id)
        post-ingest (take-while #(not= :ingestion
                                       (:document-history/change-type %))
                                rows)]
    (reduce
     (fn [acc {:document-history/keys [patch edited-by created-at]}]
       (reduce (fn [acc' op]
                 (let [path (get op "path")]
                   (if (contains? acc' path)
                     acc'
                     (assoc acc' path {:edited-by edited-by
                                       :edited-at created-at}))))
               acc
               patch))
     {}
     (reverse post-ingest))))

cd orcha && clj -X:test:silent :nses '[com.getorcha.app.http.documents.view.provenance-test]' 2>&1 | grep -A 3 -E "(FAIL|ERROR|Ran)"

clj-kondo --lint src/com/getorcha/app/http/documents/view/provenance.clj test/com/getorcha/app/http/documents/view/provenance_test.clj
git add src/com/getorcha/app/http/documents/view/provenance.clj test/com/getorcha/app/http/documents/view/provenance_test.clj
git commit -m "feat: document-provenance walker"

Task 13: editable-value converts vector paths and accepts :provenance

Files:

Find the current editable-value in components.clj (around line 542 after the merge). Replace with:

(defn editable-value
  "Wraps a display value in a span that client JS turns into an inline editor.

   path       - Clojure vector of keywords/maps, e.g.
                `[:structured-data :invoice-number]` or
                `[:structured-data :line-items {:id \"li-abc\"} :amount]`.
                Leading `:structured-data` is stripped before conversion.
   field-type - :text | :number | :currency | :date | :enum | :multiline
   opts       - {:raw-value ... :currency ... :options ... :class ...
                 :provenance {:edited-by <uuid> :edited-at <inst>}}
   display    - the hiccup currently rendered for this value."
  [path field-type {:keys [raw-value currency options class provenance] :as _opts} display]
  (let [pointer (com.getorcha.json-patch.path/clj-path->pointer path)
        classes (cond-> "editable-value"
                  (some? class)      (str " " class)
                  (some? provenance) (str " is-human-edited"))]
    [:span
     (cond-> {:class           classes
              :data-field-path pointer
              :data-field-type (name field-type)}
       (some? raw-value)  (assoc :data-raw-value (str raw-value))
       (some? currency)   (assoc :data-currency currency)
       (seq options)      (assoc :data-field-options (json/generate-string options))
       (some? provenance) (assoc :title
                                 (str "Edited at "
                                      (str (:edited-at provenance)))))
     display]))

Add to the ns form's :require:

[com.getorcha.json-patch.path]

Append to resources/app/public/css/style.css:

.editable-value.is-human-edited {
  box-shadow: inset 0 -1px 0 rgba(63, 185, 80, 0.6);
}

.edit-error-banner {
  display: block;
  margin-top: 4px;
  padding: 6px 8px;
  background-color: rgba(248, 81, 73, 0.1);
  border: 1px solid rgba(248, 81, 73, 0.4);
  border-radius: 4px;
  color: #f85149;
  font-size: 12px;
  line-height: 1.4;
}

.line-items-sortable .drag-handle {
  cursor: grab;
  color: #6e7681;
  padding: 0 8px;
}

.line-items-sortable .drag-handle:active {
  cursor: grabbing;
}

clj-kondo --lint src/com/getorcha/app/ui/components.clj

git add src/com/getorcha/app/ui/components.clj resources/app/public/css/style.css
git commit -m "feat(ui): editable-value accepts :provenance, converts vector paths"

Task 14: Thread provenance + document version through views

Files:

grep -n "detail-container\|document-detail" src/com/getorcha/app/http/documents/view/shared.clj

Find the hiccup vector that emits the outer .detail-container div.

In the shared detail-container builder:

(defn detail-container
  [{:keys [db-pool document] :as _request} body]
  (let [provenance-map (com.getorcha.app.http.documents.view.provenance/document-provenance
                        db-pool (:document/id document))]
    [:div.detail-container
     {:data-document-version (:document/version document)
      :data-document-id      (:document/id document)}
     (body provenance-map)]))

Adjust signature to match the current function; the key points are: include data-document-version and data-document-id on the wrapper, and pass provenance-map into the inner body-rendering function.

Require at the top:

[com.getorcha.app.http.documents.view.provenance]

In invoice.clj, locate the calls to invoice-header, line-items-table, line-item-card, party-card, etc. Update the signatures of those components in components.clj to accept a :provenance-map kwarg, and pass it into each edit helper's map like:

(let [edit (fn [k type raw]
             {:path      [:structured-data k]
              :type      type
              :raw-value raw
              :provenance (get provenance-map
                               (com.getorcha.json-patch.path/clj-path->pointer
                                 [:structured-data k]))})]
  ...)

The existing labeled-field / value-row / party-card already forward the :editable map as opts. Add :provenance to the merge so it reaches editable-value.

cd orcha && clj-nrepl-eval -p <PORT> "(integrant.repl/reset)"
# In a browser, open a document with edits backfilled (Task 5). Open DevTools → Inspect the field. Confirm class `is-human-edited` and a `title` attr.

clj-kondo --lint src/com/getorcha/app src/com/getorcha/app/http/documents/view
git add src/com/getorcha/app/ui/components.clj src/com/getorcha/app/http/documents/view
git commit -m "feat(ui): thread document version + provenance into detail view"

Phase 4 — Scalar edit endpoint

Task 15: Scalar edit endpoint + shared handler helpers

Files:

(ns com.getorcha.app.http.documents.edits
  "Edit endpoints for document structured_data.

   All endpoints share one core flow: check version, apply patch, write
   document_history row, update document, bump version — all in one
   transaction. Responses are HTML fragments; status code selects the
   variant and HTMX swaps whatever came back."
  (:require [cheshire.core :as json]
            [com.getorcha.app.http.documents.shared :as documents.shared]
            [com.getorcha.app.http.documents.view.provenance :as provenance]
            [com.getorcha.app.http.identity :as app.http.identity]
            [com.getorcha.app.ui.components :as ui]
            [com.getorcha.db.document-history :as dh]
            [com.getorcha.db.sql :as db.sql]
            [com.getorcha.json-patch :as jp]
            [com.getorcha.json-patch.path :as jp.path]
            [hiccup2.core :as hiccup]
            [next.jdbc :as jdbc]
            [ring.util.http-response :as ring.resp]))


(defn ^:private fetch-document-for-update
  [tx document-id le-ids]
  (db.sql/execute-one!
   tx
   {:select [:*]
    :from   [:document]
    :where  [:and
             [:= :id document-id]
             [:in :legal-entity-id le-ids]]
    :for    :update}
   {:builder-fn documents.shared/document-builder-fn}))


(defn ^:private oob-version-swap [new-version]
  [:div {:id "document-version-holder"
         :hx-swap-oob "outerHTML"
         :data-document-version new-version}])


(defn ^:private render-fragment
  [status hiccup-body triggers]
  (let [body (str (hiccup/html hiccup-body))]
    (cond-> {:status  status
             :headers {"Content-Type" "text/html; charset=utf-8"}
             :body    body}
      (seq triggers) (assoc-in [:headers "HX-Trigger"]
                               (json/generate-string triggers)))))


(defn ^:private conflict-fragment
  [path-pointer current-value current-version]
  (render-fragment
   409
   [:span.editable-value.is-rejected
    {:data-field-path path-pointer
     :data-document-version current-version}
    [:span.editable-value-current current-value]
    [:span.edit-error-banner
     "Another user modified this field. Refresh to see the latest."]
    (oob-version-swap current-version)]
   {"editRejected" true}))


(defn ^:private apply-edit-tx
  "Core edit flow. Options:
     :patch-builder    `(fn [structured-data])` returning an RFC 6902 patch vector.
     :render-success   `(fn [new-sd new-version])` returning hiccup for the
                       success fragment (before the OOB version swap is appended).
     :success-status   http status for success (defaults to 200).
     :extra-triggers   map merged into the HX-Trigger header on success (and
                       conflict) responses."
  [db-pool identity-id document-id expected-version le-ids
   {:keys [patch-builder render-success success-status extra-triggers]
    :or   {success-status 200 extra-triggers {}}}]
  (jdbc/with-transaction [tx db-pool]
    (if-let [{:document/keys [structured-data version]}
             (fetch-document-for-update tx document-id le-ids)]
      (cond
        (not= version expected-version)
        (conflict-fragment "" (pr-str structured-data) version)

        :else
        (let [patch       (patch-builder structured-data)
              new-sd      (jp/apply-patch structured-data patch)
              new-version (inc version)]
          (dh/insert! tx {:document-id document-id
                          :change-type :edit
                          :edited-by   identity-id
                          :patch       patch})
          (db.sql/execute-one!
           tx
           {:update :document
            :set    {:structured-data [:cast (json/generate-string new-sd) :jsonb]
                     :version         new-version
                     :updated-at      (java.time.Instant/now)}
            :where  [:= :id document-id]})
          (render-fragment
           success-status
           [:span
            (render-success new-sd new-version)
            (oob-version-swap new-version)]
           (merge {"documentEdited" true} extra-triggers))))
      (render-fragment 404
                       [:div.edit-error-banner "Document not found."]
                       {"toast" {:level "error" :message "Document not found."}}))))


(defn ^:private scalar-edit-handler
  [{:keys [db-pool parameters] :as request}]
  (let [{:keys [document-id]}                   (:path parameters)
        {:keys [path value expected-version]}    (:form parameters)
        identity-id                             (:identity/id (:identity request))
        le-ids                                  (mapv :legal-entity/id
                                                      (app.http.identity/legal-entities request))
        clj-path                                (jp.path/pointer->clj-path path)]
    (apply-edit-tx
     db-pool identity-id document-id
     (parse-long expected-version) le-ids
     {:patch-builder
      (fn [_sd]
        [{"op" "replace" "path" path "value" value}])
      :render-success
      (fn [_new-sd _new-version]
        ;; MVP: return a bare `editable-value` span with just the new value
        ;; as text. The swap target is the old `.editable-value` (outerHTML),
        ;; so the surrounding `.field-group` / `.value-row` markup stays intact.
        ;; The inner display class (`.field-value`, `.value-amount`, etc.) is
        ;; dropped on the edited element until page reload — acceptable
        ;; polish trade-off for MVP. Polish follow-up: render via the original
        ;; component by threading the render-context key from the client.
        (ui/editable-value
         clj-path :text
         {:provenance {:edited-by identity-id
                       :edited-at (java.time.Instant/now)}
          :class      "is-saving"}
         (str value)))})))


(defn routes [_config]
  ["/:document-id"
   ["/structured-data"
    {:name ::scalar-edit
     :patch {:summary    "Edit a scalar value in structured_data"
             :parameters {:path {:document-id :uuid}
                          :form {:path              :string
                                 :value             :string
                                 :expected-version  :string}}
             :handler    #'scalar-edit-handler}}]])

In src/com/getorcha/app/http/documents.clj add the require and route:

[com.getorcha.app.http.documents.edits :as edits]

Inside the routes fn's vector, add (edits/routes _config) alongside the existing (management/routes _config) etc.

(ns com.getorcha.app.http.documents.edits-test
  (:require [clojure.test :refer [deftest is testing use-fixtures]]
            [com.getorcha.db.sql :as db.sql]
            [com.getorcha.test.fixtures :as fixtures]))


(use-fixtures :once fixtures/system-fixture)
(use-fixtures :each fixtures/db-transaction-fixture)


(defn ^:private seed-editable-doc!
  "Inserts identity, legal-entity membership, and a document with
   version=1 and minimal structured_data. Returns
   {:doc-id, :identity}. Use `fixtures/authenticated-request`
   with the returned identity to make requests."
  []
  (let [identity (:identity (db.sql/execute-one!
                             fixtures/*db*
                             {:select [:*] :from [:identity] :limit 1}))
        le-id    (:legal-entity/id
                  (db.sql/execute-one! fixtures/*db*
                                       {:select [:id] :from [:legal-entity] :limit 1}))
        doc-id   (random-uuid)]
    (db.sql/execute-one!
     fixtures/*db*
     {:insert-into :document
      :values [{:id              doc-id
                :legal-entity-id le-id
                :content-hash    (str "hash-" doc-id)
                :file-path       "x.pdf"
                :version         1
                :structured-data [:cast (cheshire.core/generate-string
                                         {:document-type  "invoice"
                                          :invoice-number "INV-1"
                                          :line-items     []})
                                  :jsonb]}]})
    {:doc-id doc-id :identity identity}))


(defn ^:private seed-editable-doc-with-line-items!
  "Same as seed-editable-doc!, but seeds with one line item."
  []
  (let [{:keys [identity] :as seed} (seed-editable-doc!)]
    (db.sql/execute-one!
     fixtures/*db*
     {:update :document
      :set    {:structured-data
               [:cast (cheshire.core/generate-string
                       {:document-type  "invoice"
                        :invoice-number "INV-1"
                        :line-items     [{:id    (str (random-uuid))
                                          :order 0
                                          :description  "first"
                                          :page-location [1 1]}]})
                :jsonb]}
      :where [:= :id (:doc-id seed)]})
    seed))


(deftest scalar-edit-success
  (let [{:keys [doc-id identity]} (seed-editable-doc!)
        response (fixtures/authenticated-request
                  identity :patch (str "/documents/" doc-id "/structured-data")
                  {:form-params {"path"             "/invoice-number"
                                 "value"            "INV-99"
                                 "expected-version" "1"}})]
    (is (= 200 (:status response)))
    (is (re-find #"INV-99" (:body response)))
    (let [doc (db.sql/execute-one! fixtures/*db*
                                   {:select [:*] :from [:document]
                                    :where [:= :id doc-id]})]
      (is (= 2 (:document/version doc))))))


(deftest scalar-edit-conflict-on-stale-version
  (let [{:keys [doc-id identity]} (seed-editable-doc!)]
    ;; First edit — succeeds, bumps version to 2.
    (fixtures/authenticated-request
     identity :patch (str "/documents/" doc-id "/structured-data")
     {:form-params {"path" "/invoice-number" "value" "INV-2" "expected-version" "1"}})
    ;; Second edit with the old version — rejected.
    (let [response (fixtures/authenticated-request
                    identity :patch (str "/documents/" doc-id "/structured-data")
                    {:form-params {"path" "/invoice-number"
                                   "value" "INV-3"
                                   "expected-version" "1"}})]
      (is (= 409 (:status response)))
      (is (re-find #"edit-error-banner" (:body response))))))


(deftest scalar-edit-not-found
  (let [{:keys [identity]} (seed-editable-doc!)
        response (fixtures/authenticated-request
                  identity :patch (str "/documents/" (random-uuid) "/structured-data")
                  {:form-params {"path" "/invoice-number"
                                 "value" "X" "expected-version" "1"}})]
    (is (= 404 (:status response)))))

The helper seed-editable-doc! inserts an identity + legal-entity + document with a minimal structured-data (no line items — keeps the fixture small). fixtures/authenticated-request is the existing helper that wraps requests with session auth; check test/com/getorcha/test/fixtures.clj for its current signature and adapt the calls.

cd orcha && clj -X:test:silent :nses '[com.getorcha.app.http.documents.edits-test]' 2>&1 | grep -A 3 -E "(FAIL|ERROR|Ran)"

clj-kondo --lint src/com/getorcha/app/http/documents/edits.clj src/com/getorcha/app/http/documents.clj test/com/getorcha/app/http/documents/edits_test.clj
git add src/com/getorcha/app/http/documents.clj src/com/getorcha/app/http/documents/edits.clj test/com/getorcha/app/http/documents/edits_test.clj
git commit -m "feat: scalar edit endpoint with optimistic locking"

Task 16: Rewrite editable-fields.js commit path to htmx.ajax

Files:

Find the existing commit function (around line 172). Replace its body with:

function commit(input) {
  const el = input.closest('.editable-value');
  if (!el || !el.classList.contains(EDITING)) return;
  if (el.dataset.committing === '1') return;
  el.dataset.committing = '1';
  input.removeEventListener('keydown', onInputKeydown);
  input.removeEventListener('blur', onInputBlur);

  const container = document.querySelector('.detail-container');
  const documentId = container?.dataset.documentId;
  const expectedVersion = container?.dataset.documentVersion;
  const path = el.dataset.fieldPath;
  const value = readInputValue(input, el);

  if (!documentId || !path) {
    el.innerHTML = el.dataset.originalHtml;
    el.classList.remove(EDITING);
    delete el.dataset.committing;
    delete el.dataset.originalHtml;
    return;
  }

  window.htmx.ajax('PATCH', `/documents/${documentId}/structured-data`, {
    values:  { path, value, 'expected-version': expectedVersion },
    target:  el,
    swap:    'outerHTML',
    source:  el,
  }).finally(() => { delete el.dataset.committing; });
}

function readInputValue(input, el) {
  // Currency keeps `.amount` span + `.currency-suffix`; only the amount is edited.
  return input.value;
}

Remove the now-unused updateDisplayedText function — HTMX owns the DOM after commit.

Keep cancel, startEdit, builders, keyboard handler, and the listener setup unchanged.

Listen for OOB version updates. Add at the top level of the module (after the 'use strict' line):

document.body.addEventListener('htmx:afterOnLoad', (evt) => {
  const xhr = evt.detail?.xhr;
  if (!xhr) return;
  // The server may also return an updated .detail-container wrapper as an OOB swap —
  // HTMX handles it automatically. Nothing to do here beyond letting HTMX run.
});

(The OOB swap is handled entirely by HTMX's native OOB processing since the server response includes an element with hx-swap-oob.)

cd orcha && bb dev:start-server &
# In a browser: navigate to a document, click a field, change value, press Enter.
# DevTools Network tab: PATCH /documents/<id>/structured-data fires.
# Reload page: the new value persists.

git add resources/app/public/js/editable-fields.js
git commit -m "feat(ui): editable-fields commits via htmx.ajax"

Phase 5 — Line-item CRUD + reorder

Task 17: Line-item add endpoint + UI "+ Add line item" button

Files:

Append to edits.clj:

(defn ^:private add-line-item-handler
  [{:keys [db-pool parameters] :as request}]
  (let [{:keys [document-id]}        (:path parameters)
        {:keys [expected-version]}   (:form parameters)
        identity-id                  (:identity/id (:identity request))
        le-ids                       (mapv :legal-entity/id
                                           (app.http.identity/legal-entities request))
        new-id                       (str (random-uuid))]
    (apply-edit-tx
     db-pool identity-id document-id
     (parse-long expected-version) le-ids
     {:patch-builder
      (fn [sd]
        (let [next-order (->> sd :line-items (map :order) (reduce max -1) inc)]
          [{"op"    "add"
            "path"  "/line-items/-"
            "value" {"id"            new-id
                     "order"         next-order
                     "description"   "New line item"
                     "page-location" [0 0]}}]))
      :render-success
      (fn [new-sd _new-version]
        ;; Re-render only the new row. Requires extracting a `line-item-row`
        ;; helper from the existing `line-items-table` — see Step 1a below.
        (ui/line-item-row
         (->> new-sd :line-items (filter #(= new-id (:id %))) first)
         (:currency new-sd)))
      :success-status 201
      :extra-triggers {"focusNewItem" {"item-id" new-id}}})))

Extend the routes vector:

["/line-items"
 {:name ::line-items-collection
  :post {:parameters {:path {:document-id :uuid}
                      :form {:expected-version :string}}
         :handler    #'add-line-item-handler}}]

In src/com/getorcha/app/ui/components.clj, factor the existing line-items-table body into:

(defn line-item-row
  "Renders one `<tr>` for a line item. Used both by `line-items-tbody`
   and by `add-line-item-handler` to re-render the newly added row.

   The body is the existing `for`-loop body from `line-items-table`
   (the hiccup vector starting at `[:tr ...]`) with `idx` replaced by
   `(:id _item)` in the `edit` helper, and `^{:key idx}` replaced by
   `^{:key (:id _item)}`. Copy it verbatim from the current file into
   this function, then delete the original body from `line-items-table`."
  [{:keys [id] :as _item} currency]
  ;; <<copied tr hiccup goes here — see docstring>>
  )


(defn line-items-tbody
  "Renders the `<tbody>` for the sortable line-items list. Used by the
   reorder handler to return an authoritative, sorted tbody."
  [items currency {:keys [document-id]}]
  [:tbody.line-items-sortable
   {:hx-patch   (str "/documents/" document-id "/line-items")
    :hx-trigger "end"
    :hx-include "closest tbody"
    :hx-vals    "js:{'expected-version': document.querySelector('.detail-container').dataset.documentVersion}"
    :hx-target  "this"
    :hx-swap    "outerHTML"}
   (for [item (sort-by :order items)]
     (line-item-row item currency))])

Then update line-items-table to delegate to line-items-tbody. Run the test suite to ensure no regression in how documents currently render.

Near the line-items section, add a button with:

[:button.add-line-item
 {:hx-post   (str "/documents/" document-id "/line-items")
  :hx-vals   "js:{'expected-version': document.querySelector('.detail-container').dataset.documentVersion}"
  :hx-target "closest .line-items-tbody"
  :hx-swap   "beforeend"}
 [:iconify-icon {:icon "lucide:plus"}]
 " Add line item"]

Append to the top-level init section:

document.body.addEventListener('focusNewItem', (evt) => {
  const newRow = document.querySelector('.line-items-tbody > tr:last-child');
  const desc = newRow?.querySelector('.editable-value[data-field-path*="/description"]');
  desc?.click();
});

Append to edits_test.clj:

(deftest add-line-item-success
  (let [{:keys [doc-id identity]} (seed-editable-doc-with-line-items!)
        response (fixtures/authenticated-request
                  identity :post (str "/documents/" doc-id "/line-items")
                  {:form-params {"expected-version" "1"}})]
    (is (= 200 (:status response)))
    (let [doc (db.sql/execute-one! fixtures/*db*
                                   {:select [:*] :from [:document]
                                    :where [:= :id doc-id]})]
      (is (= 2 (:document/version doc)))
      (is (= 2 (count (-> doc :document/structured-data :line-items)))))))

Open a document, click "+ Add line item". New row appears at the end; description is in edit mode.

clj-kondo --lint src/com/getorcha/app/http/documents/edits.clj src/com/getorcha/app/ui/components.clj
git add src/com/getorcha/app/http/documents/edits.clj src/com/getorcha/app/ui/components.clj resources/app/public/js/editable-fields.js
git commit -m "feat: add line-item endpoint and + Add button"

Task 18: Line-item delete endpoint + per-row delete button

Files:

Append to edits.clj:

(defn ^:private remove-line-item-handler
  [{:keys [db-pool parameters] :as request}]
  (let [{:keys [document-id item-id]}   (:path parameters)
        expected-version                 (parse-long (get-in parameters [:query :expected-version]))
        identity-id                      (:identity/id (:identity request))
        le-ids                           (mapv :legal-entity/id
                                               (app.http.identity/legal-entities request))]
    (apply-edit-tx
     db-pool identity-id document-id expected-version le-ids
     {:patch-builder
      (fn [_sd]
        [{"op"   "remove"
          "path" (str "/line-items[id=" item-id "]")}])
      :render-success
      (fn [_new-sd _new-version]
        ;; Empty success body: HTMX swaps the row target with nothing.
        "")})))

Route addition:

["/line-items/:item-id"
 {:name ::line-item-by-id
  :delete {:parameters {:path  {:document-id :uuid :item-id :string}
                        :query {:expected-version :string}}
           :handler    #'remove-line-item-handler}}]

[:button.line-item-delete
 {:hx-delete    (str "/documents/" document-id "/line-items/" (:id item)
                     "?expected-version="
                     "${document.querySelector(\".detail-container\").dataset.documentVersion}")
  :hx-target    "closest tr"
  :hx-swap      "outerHTML"
  :hx-confirm   "Delete this line item?"}
 [:iconify-icon {:icon "lucide:trash-2"}]]

(If the string-interpolation approach feels brittle, use hx-vals on a POST and map it to DELETE server-side. Simpler for Reitit: keep as query param.)

(deftest delete-line-item-success
  (let [{:keys [doc-id identity]} (seed-editable-doc-with-line-items!)
        item-id (-> (db.sql/execute-one! fixtures/*db*
                                         {:select [:*] :from [:document]
                                          :where [:= :id doc-id]})
                    :document/structured-data :line-items first :id)
        response (fixtures/authenticated-request
                  identity :delete
                  (str "/documents/" doc-id "/line-items/" item-id
                       "?expected-version=1"))]
    (is (= 200 (:status response)))))

clj-kondo --lint src/com/getorcha/app/http/documents/edits.clj src/com/getorcha/app/ui/components.clj
git add src/com/getorcha/app/http/documents/edits.clj src/com/getorcha/app/ui/components.clj test/com/getorcha/app/http/documents/edits_test.clj
git commit -m "feat: delete line-item endpoint and per-row button"

Task 19: Reorder endpoint + SortableJS integration

Files:

Download and commit Sortable.min.js to resources/app/public/js/vendor/. Source: https://github.com/SortableJS/Sortable/releases (latest stable 1.x).

resources/app/public/js/sortable-glue.js:

(function () {
  'use strict';
  if (!window.htmx || !window.Sortable) return;

  window.htmx.onLoad((content) => {
    content.querySelectorAll('.line-items-sortable').forEach((el) => {
      if (el.__sortableInit) return;
      el.__sortableInit = true;
      new window.Sortable(el, {
        animation: 150,
        handle:    '.drag-handle',
      });
    });
  });
})();

In whatever hiccup builder wraps the detail view's <head> or end-of-body:

[:script {:src (str "/public/js/vendor/Sortable.min.js?v=" assets-version)}]
[:script {:src (str "/public/js/sortable-glue.js?v=" assets-version)}]

(These go alongside the existing editable-fields.js include.)

In line-items-table:

[:tbody.line-items-sortable
 {:hx-patch   (str "/documents/" document-id "/line-items")
  :hx-trigger "end"
  :hx-include "closest tbody"
  :hx-vals    "js:{'expected-version': document.querySelector('.detail-container').dataset.documentVersion}"
  :hx-target  "this"
  :hx-swap    "outerHTML"}
 (for [{:keys [id] :as item} items]
   [:tr {:data-item-id id}
    [:td.drag-handle [:iconify-icon {:icon "lucide:grip-vertical"}]]
    ;; existing cells here
    [:input {:type "hidden" :name "item-id" :value id}]])]

Append to edits.clj:

(defn ^:private reorder-line-items-handler
  [{:keys [db-pool parameters] :as request}]
  (let [{:keys [document-id]}      (:path parameters)
        {:keys [expected-version]} (:form parameters)
        ;; `item-id` appears multiple times in the form — Reitit puts it in a vector.
        item-ids                   (get-in parameters [:form :item-id])
        item-ids-vec               (if (vector? item-ids) item-ids [item-ids])
        identity-id                (:identity/id (:identity request))
        le-ids                     (mapv :legal-entity/id
                                         (app.http.identity/legal-entities request))]
    (apply-edit-tx
     db-pool identity-id document-id
     (parse-long expected-version) le-ids
     {:patch-builder
      (fn [_sd]
        (vec (map-indexed
              (fn [idx id]
                {"op"    "replace"
                 "path"  (str "/line-items[id=" id "]/order")
                 "value" idx})
              item-ids-vec)))
      :render-success
      (fn [new-sd _new-version]
        ;; Re-render the full tbody — call the hiccup component.
        (ui/line-items-tbody (:line-items new-sd)
                             (:currency new-sd)
                             {:document-id document-id}))})))

Route addition:

["/line-items"
 {:patch {:parameters {:path {:document-id :uuid}
                       :form {:expected-version :string
                              :item-id [:vector :string]}}
          :handler    #'reorder-line-items-handler}}]

(deftest reorder-line-items-success
  (let [{:keys [doc-id identity]} (seed-editable-doc-with-line-items!)
        items (-> (db.sql/execute-one! fixtures/*db*
                                       {:select [:*] :from [:document]
                                        :where [:= :id doc-id]})
                  :document/structured-data :line-items)
        reversed-ids (reverse (mapv :id items))
        response (fixtures/authenticated-request
                  identity :patch (str "/documents/" doc-id "/line-items")
                  {:form-params (into [["expected-version" "1"]]
                                      (map vector (repeat "item-id") reversed-ids))})]
    (is (= 200 (:status response)))
    (let [fresh (db.sql/execute-one! fixtures/*db*
                                     {:select [:*] :from [:document]
                                      :where [:= :id doc-id]})
          orders-by-id (into {} (map (juxt :id :order)
                                     (-> fresh :document/structured-data :line-items)))]
      (doseq [[expected-order id] (map-indexed vector reversed-ids)]
        (is (= expected-order (orders-by-id id)))))))

Open a document with 3+ line items. Drag to reorder. Refresh — order persists.

clj-kondo --lint src/com/getorcha/app/http/documents/edits.clj src/com/getorcha/app/ui/components.clj
git add src/com/getorcha/app/http/documents/edits.clj src/com/getorcha/app/ui/components.clj resources/app/public/js/ test/com/getorcha/app/http/documents/edits_test.clj
git commit -m "feat: line-item reorder with SortableJS"

Phase 6 — Downstream updates

Task 20: Update scripts/debug_fetch_document.clj and debug_common.clj

Files:

(def document-history-jsonb-keys #{:patch})

And an enum-keys map:

(def document-history-enum-keys {:change-type :document_history_change_type})

Use cast-special-fields as it's used for other tables.

Add to the {:document, :ingestions, :export-audits} map construction:

:document-history (mapv stringify-pgobjects
                        (db.sql/execute!
                         db-pool
                         {:select   [:*]
                          :from     [:document-history]
                          :where    [:= :document-id document-id]
                          :order-by [[:created-at :asc]]}))

Find where the script inserts ingestions locally. After that loop, insert history rows, using the new document-history-jsonb-keys + document-history-enum-keys for casting. Note: don't preserve the original history id (set DEFAULT uuidv7 on insert) unless the script cares — simpler to re-generate.

cd orcha && bb scripts:debug-fetch-document <some-prod-doc-id>
psql -h localhost -U postgres -d orcha -c "SELECT count(*) FROM document_history WHERE document_id = '<doc-id>'"

git add scripts/debug_common.clj scripts/debug_fetch_document.clj
git commit -m "scripts: debug-fetch-document copies document_history"

Task 21: Update the debug-doc and ingestion-regression-test skills

Files:

Find in SKILL.md line ~115-124:

psql -h localhost -U postgres -d orcha -c "SELECT structured_data FROM ap_ingestion WHERE id = '<ingestion-id>'" -x

Replace with:

# Per-ingestion extracted state (was ingestion.structured_data):
psql -h localhost -U postgres -d orcha -c \
  "SELECT patch FROM document_history WHERE ingestion_id = '<ingestion-id>' AND change_type = 'ingestion' LIMIT 1" -x

# Current materialized state (unchanged):
psql -h localhost -U postgres -d orcha -c \
  "SELECT structured_data FROM document WHERE id = '<doc-id>'" -x

# All edits since the last ingestion:
psql -h localhost -U postgres -d orcha -c \
  "WITH latest AS (SELECT MAX(created_at) AS ts FROM document_history WHERE document_id = '<doc-id>' AND change_type = 'ingestion')
   SELECT edited_by, created_at, patch FROM document_history, latest
   WHERE document_id = '<doc-id>' AND change_type = 'edit' AND created_at > latest.ts
   ORDER BY created_at" -x

Also check ap_ingestion — the current table name is ingestion per init.sql:204. If ap_ingestion is an old alias, update all references in the skill to ingestion.

Replace each of these queries:

psql -h localhost -U postgres -d orcha -c "SELECT structured_data, commit_sha FROM ingestion WHERE …"

with:

psql -h localhost -U postgres -d orcha -c \
  "SELECT (patch->0->>'value')::jsonb AS structured_data,
          (SELECT commit_sha FROM ingestion i WHERE i.id = dh.ingestion_id) AS commit_sha
   FROM document_history dh
   WHERE dh.document_id = '{{DOC_ID}}' AND dh.change_type = 'ingestion'
   ORDER BY dh.created_at DESC LIMIT 1"

(Adapt each of the three sites similarly — the baseline, the post-change state, and the comparison query all follow the same substitution pattern.)

git add .claude/skills/debug-doc/SKILL.md .claude/skills/ingestion-regression-test/inspector-prompt.md
git commit -m "skills: update debug-doc and ingestion-regression-test to read from document_history"

Verification — end to end

cd orcha && clj -X:test:silent 2>&1 | grep -E "FAIL|ERROR|Ran .* tests"

Expected: 0 failures, 0 errors.

cd orcha && clj-kondo --lint src test dev 2>&1 | tail -3

Expected: errors: 0, warnings: 0.

Against the running dev server, on a real document:

  1. Edit a scalar field (invoice number). Page shows green flash; reload → new value persists.
  2. Edit a nested scalar (party name). Same check.
  3. Edit a line-item cell (amount). Verify document_history got a new edit row with [id=X] in the path.
  4. Click "+ Add line item". New row appears; description opens in edit mode; document.version bumped.
  5. Drag to reorder. Refresh — order persists.
  6. Click a row's delete button, confirm the browser prompt. Row disappears.
  7. Simulate a conflict: open two tabs of the same document. Edit field in tab A. In tab B (stale version), edit the same field → red error banner appears, server's new value displayed.

After Step 3, inspect an edited field via DevTools → confirm class="editable-value is-human-edited" and a title="Edited at …" attribute.

Notes for the implementing worker