Stack Research

Domain: Semantic Search Comparison Benchmark (pgvector vs LLM-based matching) Researched: 2026-02-20 Confidence: HIGH

Recommended Stack

Core Technologies

Database Layer

Embedding Libraries

Embedding Models Supported

Benchmarking Framework

Interactive Dashboard

Development Tools

Docker Setup

Recommended Docker Compose

version: '3.8'
services:
  postgres:
    image: pgvector/pgvector:0.8.1-pg18
    environment:
      POSTGRES_USER: semantic
      POSTGRES_PASSWORD: semantic
      POSTGRES_DB: semantic_search
    ports:
      - "5432:5432"
    volumes:
      # IMPORTANT: Postgres 18 changed PGDATA location
      - pgdata:/var/lib/postgresql
    shm_size: 256mb  # Required for parallel HNSW index builds

volumes:
  pgdata:

pgvector Index Configuration

-- Enable extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Create table with vector column
CREATE TABLE line_items (
    id SERIAL PRIMARY KEY,
    supplier_name TEXT,
    description TEXT,
    embedding vector(768),  -- Adjust dimensions per model
    debit_account TEXT,
    credit_account TEXT,
    cost_center TEXT
);

-- HNSW index (faster queries, slower builds)
CREATE INDEX ON line_items USING hnsw (embedding vector_cosine_ops);

-- Alternative: IVFFlat (faster builds, good for prototyping)
-- CREATE INDEX ON line_items USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);

Installation

# Core dependencies
pip install psycopg[binary] pgvector google-genai sentence-transformers

# Benchmarking
pip install ragas pandas numpy

# Dashboard
pip install streamlit plotly

# Development
pip install python-dotenv pytest

# Full install (single command)
pip install psycopg[binary] pgvector google-genai sentence-transformers ragas pandas numpy streamlit plotly python-dotenv pytest

Requirements.txt

# Database
psycopg[binary]>=3.3.0
pgvector>=0.4.2

# Embeddings
google-genai>=1.64.0
sentence-transformers>=5.2.0

# Benchmarking
ragas>=0.4.0
pandas>=2.0.0
numpy>=1.24.0

# Dashboard
streamlit>=1.54.0
plotly>=5.0.0

# Development
python-dotenv>=1.0.0
pytest>=8.0.0

Alternatives Considered

What NOT to Use

Stack Patterns by Variant

For API-based embeddings (production-like):

• Use google-genai for Google models
• Use Jina API directly via requests (no official Python SDK needed)
• Measure API latency and cost per query

For local embeddings (baseline comparison):

• Use sentence-transformers with all-MiniLM-L6-v2
• No API costs, predictable latency
• Can also run Jina v3 locally via sentence-transformers

For LLM-based matching (Orcha replication):

• Use google-genai with Gemini 2.5 Flash
• Pass 50 historical bookings as CSV context
• Measure token usage for cost comparison

Version Compatibility Matrix

Recommended Python: 3.12 - Stable, supported by all libraries, good performance.

Embedding Model Selection Guide

API Key Requirements

Sources

• pgvector GitHub - v0.8.1 release notes, Docker setup (HIGH confidence)
• pgvector Docker Hub - pg18 image tags (HIGH confidence)
• Google Gen AI SDK - v1.64.0, migration guide (HIGH confidence)
• Vertex AI Deprecation Notice - June 2025 deprecation (HIGH confidence)
• sentence-transformers PyPI - v5.2.3 (HIGH confidence)
• ragas PyPI - v0.4.3 (HIGH confidence)
• streamlit PyPI - v1.54.0 (HIGH confidence)
• Jina AI Embeddings - v3/v4 models, API (MEDIUM confidence - no official PyPI package)
• Gemini Embedding Blog - Model comparison (HIGH confidence)

Stack research for: Semantic Search Comparison Benchmark Researched: 2026-02-20