Gübelin Slip Processing: Cloud Provider Cost Comparison

Project: Process 600,000 historical Warenstammkarte (inventory cards) Date: December 2025 Author: Generated for cost analysis

Executive Summary

⚠️ CRITICAL: Gemini API rate limits (RPD - Requests Per Day) are the primary constraint. Without batching, processing would take 60-600 days. Batching multiple slips per LLM call is mandatory.

Recommendation: GCP offers native integration with both Document AI OCR and Gemini Flash 2.0, reducing complexity. Costs are nearly identical between providers.

Architecture Overview

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  Source PDFs    │────▶│  Split Images   │────▶│   OCR + LLM     │
│  (~240 GB)      │     │  (~240 GB JPG)  │     │   Processing    │
└─────────────────┘     └─────────────────┘     └─────────────────┘
                                                        │
                              ┌─────────────────────────┘
                              ▼
                        ┌─────────────────┐     ┌─────────────────┐
                        │  JSON Output    │────▶│  SQLite DB      │
                        │  (~2 GB)        │     │  + Logs         │
                        └─────────────────┘     └─────────────────┘

Processing Pipeline:

1. Upload source PDFs to cloud storage
2. Spin up compute instance
3. Split PDFs into individual slip images (PyMuPDF)
4. OCR all slips (can run in parallel - not rate limited)
5. Batch LLM extraction (10 slips per call to stay within rate limits)
6. Save results to SQLite DB
7. Download results, terminate instance

⚠️ CRITICAL: API Rate Limits

Rate limits are the primary constraint on processing time, not compute or API costs.

Document AI OCR Rate Limits

Source: Document AI Quotas

Gemini API Rate Limits (THE BOTTLENECK)

Legend:

• RPM = Requests Per Minute
• TPM = Tokens Per Minute
• RPD = Requests Per Day (the limiting factor!)

Source: Gemini API Rate Limits

Rate Limit Impact Analysis

Recommendation: Use 10 slips per LLM call with Tier 2 for optimal balance of speed and reliability.

How to Unlock Tier 2

1. Enable Cloud Billing on your GCP project
2. Spend $250+ on Google Cloud services (cumulative)
3. Wait 30 days from first billing
4. Tier 2 is automatically unlocked

Source: Gemini API Rate Limits Guide

AWS Textract Rate Limits (for comparison)

Source: AWS Textract Limits

Note: Textract has no daily request limits, making it less constrained than Gemini for high-volume processing.

Detailed Cost Breakdown

1. OCR API Costs

Both providers offer equivalent OCR capabilities with word-level confidence scores.

Sources:

• Google Document AI Pricing
• AWS Textract Pricing

2. LLM API Costs (Gemini 2.0 Flash)

Gemini 2.0 Flash is used regardless of cloud provider. Based on measured token usage from test runs:

Pricing:

• Input: $0.10 per 1M tokens
• Output: $0.40 per 1M tokens

Note: Batching does not significantly change token costs (same content processed), but reduces API call overhead.

Source: Gemini API Pricing

3. Compute Costs

Comparable instances for running the Python processing pipeline:

Updated Assumptions (with rate limits):

• Processing is rate-limited by Gemini API, not compute
• Instance runs 6-12 days continuously
• Most time spent waiting for API responses

Sources:

• GCP VM Instance Pricing
• AWS EC2 On-Demand Pricing

4. Storage Costs

Notes:

• Prices for US regions (us-central1 / us-east-1)
• Delete split images after processing to reduce ongoing costs

Sources:

• Google Cloud Storage Pricing
• AWS S3 Pricing

5. Storage API Operations

6. Network Egress

Downloading final results (JSON files, SQLite DB, logs):

Note: Results are small (~3-5 GB). Both providers include 100 GB free egress monthly.

Sources:

• GCP Network Pricing
• AWS Data Transfer Pricing

7. Logging Costs

Estimated log volume: ~50-100 MB (processing logs, errors, stats)

Notes:

• Log volume is minimal (~100 MB), well under free tiers
• Logs include: per-slip timing, token usage, errors, daily summaries

Sources:

• GCP Cloud Logging Pricing
• AWS CloudWatch Pricing

Total Cost Summary

GCP (Google Cloud Platform)

AWS (Amazon Web Services)

Processing Time Estimates

Realistic Timeline (with rate limits)

Parallel Processing Breakdown

Note: OCR and LLM can run in parallel (OCR ahead of LLM), but LLM rate limits dominate the timeline.

Provider Comparison Matrix

Recommendations

Primary Recommendation: GCP with Tier 2 + Batching

Prerequisites:

1. ✅ Enable Cloud Billing (unlocks Tier 1)
2. ⏳ Spend $250+ on GCP services (unlocks Tier 2 after 30 days)
3. ✅ Implement 10-slip batching in LLM calls

Expected Timeline: 6-12 days Expected Cost: ~$1,060-$1,090

Batching Strategy (Mandatory)

Without batching:  600,000 calls → 600 days (Tier 1) or 60 days (Tier 2)
With 10x batching:  60,000 calls →  60 days (Tier 1) or  6 days (Tier 2)

The gubelin_parse.py script already includes extract_batch_with_llm() function for this purpose.

Cost Optimization Opportunities

Risk Mitigation

1. Implement checkpointing: Save progress to SQLite DB to resume after failures
2. Exponential backoff: Handle 429 rate limit errors gracefully
3. Error logging: Track failed slips for manual review
4. Cost alerts: Set up billing alerts at $500, $800, $1,000
5. Daily progress monitoring: Log daily throughput to detect issues early

Appendix A: Pricing Sources

Appendix B: Rate Limit Sources

Appendix C: Quick Reference

Timeline Summary

Cost Summary

Report generated for Gübelin historical document processing project