Plan 2: Multi-Branch on Tolaria

Date: 2026-03-20 (revised) Status: Draft Depends on: Plan 1

Summary

Run multiple orcha branches simultaneously on Tolaria (RPi 5, 8GB RAM), each accessible at {app,admin,link}-<branch>.tdev.getorcha.com. Shared PostgreSQL (one DB per branch + tolaria DB for branch registry), separate LocalStack container per branch, host JVM per branch from git worktrees. PDFs served via NPM sub_filter + restricted /_s3/ proxy (same pattern as Plan 1 — LocalStack never exposed). Branch lifecycle managed by scripts, processes managed by systemd template services.

Goals

• Run 3-4 concurrent orcha branches on Tolaria
• Each branch fully isolated: own DB, own LocalStack, own JVM, own URLs
• LocalStack never exposed to the internet (same security model as Plan 1)
• Single-command branch provisioning and teardown
• Automated NPM proxy host creation per branch
• Systemd-managed services for reliability
• Short-lived branches: slot recycling, clean teardown

Non-Goals

• CI/CD integration (branches are manually created/updated)
• Auto-scaling or load balancing
• Shared LocalStack resources across branches (decided against — isolation outweighs RAM savings)

Architecture

Network Topology

Browser
  │ HTTPS
  ▼
DNS (Route53)
  *.tdev.getorcha.com → home IP
  │ :443
  ▼
Nginx Proxy Manager
  TLS (Let's Encrypt) · Basic auth
  3 proxy hosts per branch (auto-created)
  sub_filter + /_s3/ on each app proxy host
  │ HTTP to per-branch ports
  ▼
┌──────────────────────────────────────────────────┐
│ Tolaria                                          │
│                                                  │
│  ┌─ Branch: feature-ocr (slot 1) ────────────┐  │
│  │  JVM  :18001 app  :17001 admin  :19001 link│  │
│  │  LocalStack container  :14001              │  │
│  │  DB: orcha_feature_ocr                     │  │
│  │  Worktree: spikes/tolaria/.worktrees/      │  │
│  │            feature-ocr/                    │  │
│  └────────────────────────────────────────────┘  │
│                                                  │
│  ┌─ Branch: fix-export (slot 2) ─────────────┐  │
│  │  JVM  :18002 app  :17002 admin  :19002 link│  │
│  │  LocalStack container  :14002              │  │
│  │  DB: orcha_fix_export                      │  │
│  │  Worktree: spikes/tolaria/.worktrees/      │  │
│  │            fix-export/                     │  │
│  └────────────────────────────────────────────┘  │
│                                                  │
│  PostgreSQL 18 (shared container)  :5432         │
│    DBs: orcha, tolaria, orcha_feature_ocr, ...   │
└──────────────────────────────────────────────────┘

Per-Branch Resources

Port Allocation

Each branch gets a slot number (recycled from lowest available). Ports are deterministic:

The demo instance (Plan 1) sits outside this allocation scheme — it uses default ports (8888, 7777, 9999, 4566) and is not assigned a slot.

Slot Recycling

Branches are short-lived and destroyed often. Slots are recycled — when a branch is destroyed, its slot becomes available for the next branch-create. The tolaria database tracks slot allocation (see Branch Registry below).

Domains

Three domains per branch (no S3 subdomain). All covered by the *.tdev.getorcha.com wildcard CNAME.

Other Resources

branch_safe converts hyphens to underscores for valid PostgreSQL database names.

PDF Serving (same pattern as Plan 1)

Each branch's app proxy host gets the sub_filter + /_s3/ advanced config, pointing to that branch's LocalStack port:

NPM Advanced Configuration on app-{branch}.tdev.getorcha.com:

# Rewrite LocalStack presigned URLs in HTML responses
sub_filter_once off;
sub_filter_types *;
sub_filter 'http://localhost:{14000+slot}' '/_s3';

# Serve PDFs from branch's LocalStack — restricted to document PDF paths only
location ~ ^/_s3/(v1-orcha-global-storage-[^/]+/documents/[0-9a-f-]+(/(display|email))?\.pdf)$ {
    limit_except GET { deny all; }
    rewrite ^/_s3/(.*)$ /$1 break;
    proxy_pass http://192.168.2.185:{14000+slot};
}

Implementation notes from Plan 1:

• sub_filter_types * is required (not text/html) because orcha's Jetty does not send a Content-Type header on some HTML responses.
• proxy_pass must use Tolaria's LAN IP (192.168.2.185), not 127.0.0.1, because NPM runs on a different machine (192.168.2.117).
• The rewrite directive strips the /_s3 prefix before proxying to LocalStack.
• The /_s3/ location bypasses basic auth — by design (iframe same-origin, regex restricts to document UUIDs).

Branch Registry (PostgreSQL)

A tolaria database in the shared PostgreSQL instance stores branch metadata and slot allocations.

Schema

CREATE TABLE branches (
    name       TEXT PRIMARY KEY,
    slot       INTEGER NOT NULL UNIQUE,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

Slot Allocation

-- Find lowest available slot
SELECT s FROM generate_series(1, 999) s
WHERE s NOT IN (SELECT slot FROM branches)
ORDER BY s LIMIT 1;

Operations

-- Create
INSERT INTO branches (name, slot) VALUES ('feature-ocr', 1);

-- List
SELECT * FROM branches ORDER BY slot;

-- Destroy
DELETE FROM branches WHERE name = 'feature-ocr';

The branch-create.sh script creates the tolaria DB and branches table if they don't exist (idempotent).

Config Changes (:temp-branch profile)

Plan 1 already unified bucket/DB config via #orcha/param and added the :demo profile. Plan 2 adds a :temp-branch profile. Branches run with ENVIRON_NAME=temp-branch.

Keys with :temp-branch entries

Per-branch via #env:

;; AWS endpoint — per-branch LocalStack port
:endpoint #profile {#{:local-dev :demo}       "http://localhost:4566"
                    :sandbox                   "http://localstack:4566"
                    :temp-branch               #env ORCHA_AWS_ENDPOINT
                    :default                   nil}

;; Base URLs — per-branch domains
:com.getorcha/app-base-url
#profile {:local-dev    "https://orcha.barreto.tech"
          :demo         "https://app.demo.getorcha.com"
          :temp-branch  #env ORCHA_APP_BASE_URL
          :default      "https://app.getorcha.com"}

:com.getorcha/link-base-url
#profile {:local-dev    "https://link.orcha.barreto.tech"
          :demo         "https://link.demo.getorcha.com"
          :temp-branch  #env ORCHA_LINK_BASE_URL
          :default      "https://link.getorcha.com"}

;; HTTP server ports
:port #profile {:temp-branch #long #env ORCHA_APP_PORT  :default 8888}  ;; app
:port #profile {:temp-branch #long #env ORCHA_ADMIN_PORT :default 7777} ;; admin
:port #profile {:temp-branch #long #env ORCHA_LINK_PORT  :default 9999} ;; link

Note: #long #env converts the env var string to a long. If Aero doesn't support this nesting, fall back to a custom reader #orcha/env-long.

Via #orcha/param (already unified in Plan 1 — no changes needed):

• :db-credentials-json — init_aws.clj seeds with branch-specific DB name
• S3 bucket names — init_aws.clj seeds /v1-orcha/account-id = "local-stack"
• All API keys, OAuth secrets — init_aws.clj seeds same dev values

Hardcoded :temp-branch values (same as :local-dev):

All entries that currently use #{:local-dev :demo} become #{:local-dev :demo :temp-branch}:

• :dev? → true
• :dev-file-store → {:protocol "file" :path "dump/finances"}
• :integrations :datev → sandbox endpoints
• :search :credentials-file → "credentials/google-docai-dev.json"
• :with-seed-data? → true
• :session :secure? → false
• :transcription :credentials-file → "credentials/google-docai-dev.json"
• :max-retry-attempts → 1
• :shutdown-check-ms → 0
• :wait-time-seconds (matching) → 0

Per-Branch Environment File

# Auto-generated by branch-create.sh
ENVIRON_NAME=temp-branch

# Ports
ORCHA_APP_PORT=18001
ORCHA_ADMIN_PORT=17001
ORCHA_LINK_PORT=19001

# External URLs
ORCHA_APP_BASE_URL=https://app-feature-ocr.tdev.getorcha.com
ORCHA_LINK_BASE_URL=https://link-feature-ocr.tdev.getorcha.com

# Internal endpoint
ORCHA_AWS_ENDPOINT=http://localhost:14001

# DB credentials seeded in LocalStack by init_aws.clj
# via ORCHA_DB_NAME=orcha_feature_ocr

Init Script for Temp Branches

Each branch runs the existing bb dev:init-aws against its own LocalStack. The init script already supports ORCHA_DB_HOST and ORCHA_DB_NAME env vars (added in Plan 1). For temp branches:

ORCHA_DB_NAME=orcha_feature_ocr \
ORCHA_AWS_ENDPOINT=http://localhost:14001 \
  bb dev:init-aws

This seeds:

• /v1-orcha/account-id = "local-stack" (bucket suffix — same for all branches, each has own LocalStack)
• /v1-orcha/db-credentials = {"host":"localhost","port":5432,"dbname":"orcha_feature_ocr",...}
• All OAuth/API secrets with dev values

Note: init_aws.clj reads :endpoint from config.edn via Aero. With ENVIRON_NAME=temp-branch and ORCHA_AWS_ENDPOINT=http://localhost:14001, the config resolves correctly. Must verify during implementation that Babashka's Aero correctly resolves #env in this context.

Systemd Template Service

# /etc/systemd/system/orcha@.service
[Unit]
Description=Orcha (%i)
After=network.target docker.service
Requires=docker.service

[Service]
Type=simple
User=karn
WorkingDirectory=/home/karn/orcha/spikes/tolaria/.worktrees/%i/orcha
EnvironmentFile=/home/karn/orcha/spikes/tolaria/.worktrees/%i/.env
ExecStartPre=/bin/sh -c 'docker start localstack-%i 2>/dev/null || echo "Warning: container localstack-%i not found, run branch-create.sh"'
ExecStart=/usr/bin/java -Xmx1g -jar target/orcha.jar -m com.getorcha.system
ExecStopPost=-/usr/bin/docker stop localstack-%i
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Branch Lifecycle Scripts

branch-create.sh <branch-name>

Full provisioning in one command:

1. Ensure tolaria DB exists — createdb tolaria (idempotent), create branches table if not exists
2. Allocate slot — SELECT min(s) FROM generate_series(1,999) s WHERE s NOT IN (SELECT slot FROM branches)
3. Insert registry record — INSERT INTO branches (name, slot) VALUES (...) (fails on duplicate = branch already exists)
4. Create git worktree — git worktree add spikes/tolaria/.worktrees/{branch} origin/{branch}
5. Create PostgreSQL database — createdb orcha_{branch_safe}
6. Start LocalStack container — docker run -d --name localstack-{branch} -p {14000+slot}:4566 -e PERSISTENCE=1 -e SERVICES=s3,sqs,secretsmanager,ssm -e DEFAULT_REGION=eu-central-1 -v localstack-{branch}:/var/lib/localstack localstack/localstack:latest
7. Wait for LocalStack health
8. Generate .env file
9. Init LocalStack resources — source .env && ORCHA_DB_NAME=orcha_{branch_safe} bb dev:init-aws
10. Build uberjar — cd .worktrees/{branch}/orcha && clojure -X:build
11. Run DB migrations — source .env && bb migrate migrate
12. Create NPM proxy hosts — npm-setup.sh --branch {branch} --slot {slot}
13. Enable and start systemd service — sudo systemctl enable --now orcha@{branch}

branch-destroy.sh <branch-name>

1. sudo systemctl disable --now orcha@{branch}
2. npm-setup.sh --branch {branch} --teardown
3. docker rm -f localstack-{branch} + docker volume rm localstack-{branch}
4. dropdb orcha_{branch_safe}
5. git worktree remove spikes/tolaria/.worktrees/{branch}
6. DELETE FROM branches WHERE name = '{branch}' (frees slot)

branch-update.sh <branch-name>

1. cd spikes/tolaria/.worktrees/{branch} && git pull
2. cd orcha && clojure -X:build
3. source ../.env && ORCHA_DB_NAME=orcha_{branch_safe} bb dev:init-aws
4. source ../.env && bb migrate migrate
5. sudo systemctl restart orcha@{branch}

Resource Budget (8GB RPi 5)

At 4 branches: ~6.9GB used. Consider -Xmx768m for more headroom.

File Structure

spikes/tolaria/
├── docs/
│   ├── 2026-03-19-plan1-single-instance-design.md
│   └── 2026-03-19-plan2-multi-branch-design.md
├── plans/
│   ├── 2026-03-19-plan1-single-instance.md
│   └── 2026-03-19-plan2-multi-branch.md
├── scripts/
│   ├── install-prereqs.sh      # (Plan 1)
│   ├── npm-setup.sh            # (Plan 1, extended for --branch)
│   ├── run-demo.sh             # (Plan 1)
│   ├── branch-create.sh        # (Plan 2)
│   ├── branch-destroy.sh       # (Plan 2)
│   └── branch-update.sh        # (Plan 2)
├── config/
│   └── demo.env                # (Plan 1)
├── systemd/
│   └── orcha@.service          # (Plan 2)
└── .worktrees/                 # (Plan 2, gitignored)
    ├── feature-ocr/
    │   ├── .env
    │   └── orcha/
    └── fix-export/
        ├── .env
        └── orcha/

NPM Automation (extended from Plan 1)

The npm-setup.sh script from Plan 1 already supports --branch and --slot flags:

• Without flags: creates demo proxy hosts (*.demo.getorcha.com)
• With --branch feature-ocr --slot 1: creates branch proxy hosts (*-feature-ocr.tdev.getorcha.com)
• With --branch feature-ocr --teardown: removes branch proxy hosts

Each branch gets 3 proxy hosts (app, admin, link). The app host includes sub_filter + /_s3/ advanced config. Basic auth applied to all three.

Relationship to Plan 1

The demo instance (Plan 1) remains separate:

• Uses default ports (8888/7777/9999/4566)
• Uses *.demo.getorcha.com domains
• Uses ENVIRON_NAME=demo
• Runs from the main repo checkout, not a worktree
• Not managed by systemd (started manually via run-demo.sh)

Plan 2 branches use *.tdev.getorcha.com, allocated ports, and ENVIRON_NAME=temp-branch. They share the same PostgreSQL container and NPM basic auth access list as the demo.

PostgreSQL container management: Started by Plan 1's docker-compose up -d. branch-create.sh verifies PostgreSQL is up (via pg_isready) before proceeding.

Open Items for Implementation

1. Verify bb dev:init-aws with ENVIRON_NAME=temp-branch — does Babashka's Aero resolve #env ORCHA_AWS_ENDPOINT correctly?
2. Verify #long #env nesting in Aero — does #long #env ORCHA_APP_PORT work, or do we need #orcha/env-long?
3. JVM heap tuning — start with -Xmx1g, monitor, consider -Xmx768m.
4. .worktrees/ in .gitignore — add spikes/tolaria/.worktrees/ to .gitignore.