Requirements: Semantic Search Comparison

Defined: 2026-02-20 Core Value: Determine whether pgvector semantic search can match or exceed LLM-based matching for GL account/cost center assignment

v1 Requirements

Infrastructure

INFRA-01: Docker Compose setup with Postgres 18 and pgvector 0.8.1
INFRA-02: Python environment with google-genai, sentence-transformers, ragas
INFRA-03: Import ~6K line items from historical.csv into pgvector table
INFRA-04: Pre-compute embeddings with Google text-multilingual-embedding-002
INFRA-05: Pre-compute embeddings with Jina embeddings
INFRA-06: Pre-compute embeddings with all-MiniLM-L6-v2 (local)
INFRA-07: Store embedding model metadata alongside vectors

Search

SRCH-01: Single query text input interface
SRCH-02: Top-K retrieval with adjustable K (3, 5, 10)
SRCH-03: Display similarity scores per result
SRCH-04: Side-by-side comparison view across all 3 embedding models
SRCH-05: LLM context matching (replicate Orcha approach with Gemini 2.5 Flash)
SRCH-06: Display results with supplier, description, GL accounts, cost center

Evaluation

EVAL-01: Train/test split (80/20) with held-out test set
EVAL-02: Generate synthetic query variations (synonyms, reordering, typos)
EVAL-03: Exact match accuracy for GL account assignment
EVAL-04: Exact match accuracy for cost center assignment
EVAL-05: Top-K accuracy (correct answer in top 3/5/10)
EVAL-06: Latency measurement per query (ms)
EVAL-07: Batch benchmarking over full test set
EVAL-08: LLM-as-judge evaluation for semantic relevance
EVAL-09: Cost tracking per query (tokens, API calls, estimated $)

Reporting

REPT-01: Interactive HTML dashboard for search exploration
REPT-02: Hand-picked example showcase (curated searches)
REPT-03: Static HTML report export with aggregate metrics
REPT-04: Confusion matrix for GL account prediction errors

v2 Requirements

Advanced Metrics

ADVN-01: nDCG and MRR ranking-aware metrics
ADVN-02: Score distribution histograms per model
ADVN-03: Statistical significance testing between approaches

Enhanced Visualization

VIZN-01: Interactive charts (latency distribution, accuracy by supplier)
VIZN-02: Model confidence calibration plots

Out of Scope

Feature	Reason
Real-time embedding generation	Masks search performance; pre-compute instead
User authentication	Single-user spike; no multi-tenant need
Production deployment	Spike for decision-making, not deployment
Model fine-tuning	Compare off-the-shelf first; separate spike
Continuous monitoring	No production traffic; point-in-time evaluation

Traceability

Requirement	Phase	Status
INFRA-01	Phase 1	Complete
INFRA-02	Phase 1	Complete
INFRA-03	Phase 1	Complete
INFRA-04	Phase 2	Complete
INFRA-05	Phase 2	Complete
INFRA-06	Phase 2	Complete
INFRA-07	Phase 2	Complete
EVAL-01	Phase 2	Complete
EVAL-02	Phase 2	Complete
SRCH-01	Phase 3	Complete
SRCH-02	Phase 3	Complete
SRCH-03	Phase 3	Complete
SRCH-05	Phase 3	Complete
SRCH-06	Phase 3	Complete
SRCH-04	Phase 4	Complete
EVAL-03	Phase 4	Complete
EVAL-04	Phase 4	Complete
EVAL-05	Phase 4	Complete
EVAL-06	Phase 4	Complete
EVAL-07	Phase 4	Complete
EVAL-09	Phase 4	Complete
REPT-01	Phase 4	Complete
EVAL-08	Phase 5	Complete
REPT-02	Phase 5	Complete
REPT-03	Phase 5	Complete
REPT-04	Phase 5	Complete

Coverage:

v1 requirements: 26 total
Mapped to phases: 26
Unmapped: 0

Requirements defined: 2026-02-20 Last updated: 2026-02-20 after roadmap creation