Pipeline Correction Tracking - Product Specification

Author: Product Team Status: Draft Last Updated: 2026-02-04

1. Problem Statement

When an invoice is processed through Orcha's ingestion pipeline, the initial LLM extraction is followed by deterministic validation checks that may flag errors or uncertainties. Post-processors like the UncertainValidationsResolver then re-examine the original PDF with an LLM to correct these issues — fixing math errors, removing misextracted line items, correcting tax IDs.

Currently, these corrections are applied silently to the structured data. The LLM's reasoning and confidence are stored in validation-results, but there is no record of what specific fields were modified, what the original values were, or which processor made the change.

This creates a blind spot:

• Debugging is guesswork: When a field has an unexpected value, there's no way to tell if it came from extraction or was modified by a post-processor
• No prompt improvement signal: We can't measure how often corrections happen or which fields are corrected most frequently — data that would help improve extraction prompts and reduce the need for corrections
• No data lineage: The evolution of structured data through the pipeline is not traceable

2. Goal

Add a lightweight correction log that records every modification made by corrective post-processors, enabling developers to:

1. Trace data lineage — for any field, know whether it came from extraction or was corrected, and by whom
2. Identify correction patterns — query which fields are corrected most often, by which processors, to guide extraction prompt improvements
3. Monitor pipeline health — measure what percentage of ingestions require corrections over time

3. Success Metrics

4. Core Concepts

4.1 Corrective vs. Enrichment Post-Processors

Not all post-processors modify existing data. Some add entirely new fields (enrichments). Only processors that change values originally set by extraction are in scope.

Verified: No other pipeline stage mutates extracted fields. with-validations only adds :validation-results, with-fraud-detection only adds :fraud-flags, and no transformations occur between extraction and post-processing or between post-processing and persistence.

4.2 Complete Field Inventory

Every field that can be corrected by a post-processor, verified against the codebase:

UncertainValidationsResolver — field corrections (via allowlist):

UncertainValidationsResolver — line-item corrections:

TaxComplianceAnalyzer — tax ID corrections:

4.3 Correction Action Types

Each logged correction is typed by the kind of change it represents:

4.3 Before/After Capture

Every correction logs the old value (what extraction produced) and the new value (what the post-processor set), along with the LLM's reasoning and confidence score. This enables full reconstruction of the pre-correction state without needing pipeline snapshots.

4.4 Separate Storage

Correction logs are stored in a dedicated table, separate from the structured data itself. This keeps business data clean while enabling cross-ingestion analytics (e.g., "which fields are corrected most often across all tenants?").

5. Audience

This feature is developer-only. No UI changes are introduced.

• Developers: Query the correction log via SQL for debugging and analytics
• No admin or end-user access: Corrections are not surfaced in the invoice review UI
• Future possibility: A dashboard or UI indicators could be built on top of this data later

6. Use Cases

6.1 Debugging an Unexpected Field Value

Scenario: Invoice #12345 shows subtotal = 105.0, but the PDF clearly states 100.0.
Developer queries: SELECT * FROM ingestion_correction_log
                   WHERE ingestion_id = '<id>' AND field_path = 'subtotal'
Result: Shows that UncertainValidationsResolver changed subtotal from 100.0 to 105.0
        with reasoning "Found shipping fee included in subtotal on PDF"
Action: Developer investigates whether the correction was appropriate or a model error

6.2 Identifying Extraction Improvement Opportunities

Scenario: Developer wants to know if extraction prompts should be improved.
Developer queries: SELECT field_path, action, count(*) as frequency
                   FROM ingestion_correction_log
                   WHERE tenant_id = '<id>' AND created_at > now() - interval '30 days'
                   GROUP BY field_path, action
                   ORDER BY frequency DESC
Result: line-item removals account for 40% of all corrections
Action: Improve extraction prompt to avoid extracting section subtotals as line items

6.3 Pipeline Health Monitoring

Scenario: Developer wants to measure correction rate over time.
Developer queries: SELECT
                     count(DISTINCT icl.ingestion_id)::float / count(DISTINCT i.id) as correction_rate
                   FROM ingestion i
                   LEFT JOIN ingestion_correction_log icl ON icl.ingestion_id = i.id
                   WHERE i.status = 'completed'
                     AND i.completed_at > now() - interval '30 days'
Result: 12% of ingestions required at least one correction
Action: Track this metric monthly to measure extraction quality improvements

6.4 Tracing a Tax ID Correction

Scenario: A supplier's tax ID type was reclassified from "vat" to "ein".
Developer queries: SELECT * FROM ingestion_correction_log
                   WHERE ingestion_id = '<id>' AND field_path LIKE 'issuer.tax-id%'
Result: Two corrections logged:
        - field_path: 'issuer.tax-id-type', old: "vat", new: "ein"
        - field_path: 'issuer.tax-id', old: "DE12345678", new: "12-3456789"
        Processor: tax-compliance-analyzer
Action: Developer verifies the reclassification was correct

7. Data Model

7.1 Correction Log Table

Table: ingestion_correction_log

7.2 Indexes

7.3 Example Rows

Field correction (subtotal):

Line item removal:

Tax ID type reclassification:

8. Edge Cases & Business Rules

9. Data Retention

• No auto-cleanup: Correction logs are retained indefinitely. The table is narrow (one row per corrected field) and storage cost is minimal.
• Future option: A TTL-based cleanup can be added later if the table grows large. The separate table design makes this straightforward.
• No archival: No cold storage or partitioning needed initially.

10. Out of Scope (v1)

• UI indicators: No visual markers on corrected fields in the invoice review UI
• Enrichment tracking: Post-processors that add new fields (account matching, cost centers, accruals) are not logged
• Real-time alerts: No notifications when corrections happen
• Dashboard: No built-in analytics dashboard — developers use SQL directly
• Correction approval workflow: No human-in-the-loop for reviewing corrections before they're applied
• Full pipeline snapshots: No storing of complete structured-data at each pipeline stage
• Correction undo: No mechanism to revert a correction to its original value

11. Open Questions

1. Field path format: Should field paths use dot notation (issuer.country) or vector notation ([:issuer :country])? Dot notation is more SQL-friendly; vector notation is more Clojure-idiomatic. Recommendation: dot notation for queryability.
2. Confidence threshold logging: Should we also log cases where the LLM analyzed a field but decided NOT to correct it (confidence too low or value confirmed correct)? This data could be useful for understanding false-positive rates. Recommendation: defer to v2.
3. Cross-processor conflicts: If both UncertainValidationsResolver and TaxComplianceAnalyzer try to correct the same field (e.g., tax-id), who wins? Currently they run in parallel and results are applied sequentially. The last writer wins, but both corrections would be logged. Recommendation: document this behavior and accept it for v1.