Plan 2: Multi-Branch on Tolaria — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Run multiple orcha branches simultaneously on Tolaria, each accessible at {app,admin,link}-<branch>.tdev.getorcha.com, with single-command provisioning and teardown.

Architecture: Each branch gets a git worktree, a JVM process on allocated ports, a dedicated LocalStack container, and its own PostgreSQL database. Shared PostgreSQL container. Systemd template services manage processes. NPM proxy hosts are automated via the npm-setup.sh script (extended from Plan 1). sub_filter + /_s3/ pattern keeps LocalStack unexposed.

Tech Stack: Bash, Docker, systemd template units, git worktrees, NPM REST API, jq, pass

Spec: spikes/tolaria/docs/2026-03-19-plan2-multi-branch-design.md Depends on: Plan 1 completed

Lessons from Plan 1 Implementation

These were discovered during Plan 1 and must be accounted for in Plan 2:

sub_filter_types * — orcha's Jetty doesn't send Content-Type on some HTML responses. Using text/html silently skips rewriting. Use * instead.
proxy_pass must use Tolaria's LAN IP — NPM runs on 192.168.2.117, not Tolaria. 127.0.0.1 in proxy_pass resolves to the NPM host, not Tolaria. Use 192.168.2.185.
rewrite directive needed — proxy_pass inside a regex location passes the original URI (including /_s3/ prefix) to the upstream. Add rewrite ^/_s3/(.*)$ /$1 break; to strip it.
Debian Trixie uses docker-compose (standalone) — not docker compose plugin. The docker-compose-plugin package doesn't exist in Trixie repos.
AWS CLI is a prerequisite — init_aws.clj shells out to the aws CLI. Not installed by default.
NPM cert API — meta field only accepts {"dns_challenge": false}. Passing letsencrypt_agree or letsencrypt_email causes a 400 error (additionalProperties violation).
NPM user permissions — the API user needs: Proxy Hosts: Manage, Access Lists: Manage, Certificates: Manage. "View Only" on access lists causes POST to return 404 (not 403).
/_s3/ location bypasses basic auth — this is by design (iframe same-origin loads). The regex restricts to document PDF paths only.

File Map

File	Action	Purpose
`orcha/resources/com/getorcha/config.edn`	Modify	Add `#or [#env ...]` for ports, DB credentials, AWS endpoint
`spikes/tolaria/scripts/npm-setup.sh`	Already created (Plan 1)	Supports `--branch`/`--slot` flags already
`spikes/tolaria/scripts/branch-create.sh`	Create	Full branch provisioning
`spikes/tolaria/scripts/branch-destroy.sh`	Create	Full branch teardown
`spikes/tolaria/scripts/branch-update.sh`	Create	Pull, rebuild, restart a branch
`spikes/tolaria/systemd/orcha@.service`	Create	Systemd template unit for branches

Task 1: Modify config.edn — port and DB env var overrides

Files:

Modify: orcha/resources/com/getorcha/config.edn

The exact line numbers for port configs and DB credentials need to be found at implementation time. Search for the hardcoded port values (8888, 7777, 9999) and the db-credentials-json key.

Step 1: Find port config locations

Run: grep -n '8888\|7777\|9999' orcha/resources/com/getorcha/config.edn Identify the three port definitions for app, admin, and link servers.

Step 2: Wrap app port with #or [#env ...]

Change from:

:port 8888

To:

:port #long #or [#env ORCHA_APP_PORT "8888"]

Note: #env returns a string (or nil). #or picks the first truthy value (either the env var string or the default string "8888"). #long converts the resulting string to a long. If the HTTP server already coerces strings to ints (check during implementation), the #long wrapper may be unnecessary — but it's safer to include it.

If #long #or [...] doesn't work (Aero may not support nested reader tags in this order), the fallback is to define a custom Aero reader in system.clj that reads an env var as a long:

(defmethod aero/reader 'orcha/env-long [_ _ [env-var default]]
  (Long/parseLong (or (System/getenv env-var) (str default))))

Then use: :port #orcha/env-long [ORCHA_APP_PORT 8888]

Step 3: Wrap admin port with #or [#env ...]

Same pattern for the admin server port (7777):

:port #long #or [#env ORCHA_ADMIN_PORT "7777"]

Step 4: Wrap link port with #or [#env ...]

Same for link server port (9999):

:port #long #or [#env ORCHA_LINK_PORT "9999"]

Step 5: Find and wrap DB credentials

Run: grep -n 'db-credentials' orcha/resources/com/getorcha/config.edn

Wrap with:

:db-credentials-json #or [#env ORCHA_DB_CREDENTIALS
                          #profile {:local-dev "{\"host\":\"localhost\",\"port\":5432,\"dbname\":\"orcha\",\"username\":\"postgres\"}"
                                    ...existing values...}]

Step 6: Find and wrap AWS endpoint

Run: grep -n ':endpoint' orcha/resources/com/getorcha/config.edn

Wrap the existing :endpoint key:

:endpoint #or [#env ORCHA_AWS_ENDPOINT
               #profile {:local-dev "http://localhost:4566"
                         :sandbox   "http://localstack:4566"
                         :default   nil}]

Step 7: Verify unchanged behavior

Test that without any new env vars set, config resolves to original values. Use the same Clojure one-liner approach from Plan 1 Task 2 Step 5, checking port values and DB credentials.

Step 8: Commit

git add orcha/resources/com/getorcha/config.edn
git commit -m "feat(config): allow env var override for ports, DB credentials, and AWS endpoint"

Task 2: Verify init_aws.clj reads ORCHA_AWS_ENDPOINT

Files:

Read: orcha/scripts/init_aws.clj (line 102)
Step 1: Confirm endpoint resolution

Read line 102 of init_aws.clj:

(def endpoint (get-in config [:com.getorcha/aws :config :endpoint]))

The config is loaded via Aero, which respects the #or [#env ORCHA_AWS_ENDPOINT ...] wrapper from Task 1. When ORCHA_AWS_ENDPOINT is set, this line will resolve to the env var value (e.g., http://localhost:14001). No modification needed.

Step 2: Test with env var set

ORCHA_AWS_ENDPOINT=http://localhost:14099 \
  bb -e '(load-file "scripts/init_aws.clj") (println "endpoint:" endpoint)'

Expected: endpoint: http://localhost:14099

If this fails (e.g., init_aws.clj hardcodes the endpoint), a small modification is needed. Document the fix here.

Step 3: Commit if any changes were needed

git add orcha/scripts/init_aws.clj
git commit -m "fix(init-aws): respect ORCHA_AWS_ENDPOINT env var"

Task 3: Create systemd template unit

Files:

Create: spikes/tolaria/systemd/orcha@.service
Step 1: Write the unit file

[Unit]
Description=Orcha (%i)
After=network.target docker.service
Requires=docker.service

[Service]
Type=simple
User=karn
WorkingDirectory=/home/karn/orcha-branches/%i/orcha
EnvironmentFile=/home/karn/orcha-branches/%i/.env
ExecStartPre=/bin/sh -c 'docker start localstack-%i 2>/dev/null || echo "Warning: container localstack-%i not found, run branch-create.sh"'
ExecStart=/usr/bin/java -Xmx1g -jar target/orcha.jar -m com.getorcha.system
ExecStopPost=-/usr/bin/docker stop localstack-%i
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Step 2: Install the unit file

sudo cp spikes/tolaria/systemd/orcha@.service /etc/systemd/system/orcha@.service
sudo systemctl daemon-reload

Step 3: Verify systemd recognizes it

Run: systemctl cat orcha@test 2>&1 | head -5 Expected: shows the unit file content (even though "test" instance doesn't exist yet)

Step 4: Commit

git add spikes/tolaria/systemd/orcha@.service
git commit -m "feat(tolaria): add systemd template unit for orcha branches"

Task 4: Create branch-create.sh

Files:

Create: spikes/tolaria/scripts/branch-create.sh
Step 1: Write branch-create.sh

#!/usr/bin/env bash
set -euo pipefail

# Provision a new orcha branch environment on Tolaria.
# Usage: branch-create.sh <branch-name>
#
# Creates: git worktree, PostgreSQL DB, LocalStack container,
#          .env file, NPM proxy hosts, systemd service.

if [[ $# -lt 1 ]]; then
  echo "Usage: branch-create.sh <branch-name>"
  exit 1
fi

BRANCH="$1"
BRANCH_SAFE="${BRANCH//-/_}"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
BRANCHES_DIR="$HOME/orcha-branches"
BRANCH_DIR="${BRANCHES_DIR}/${BRANCH}"
REGISTRY="${BRANCHES_DIR}/registry.json"

echo "=== Creating branch environment: ${BRANCH} ==="

# ── 1. Allocate slot ──
mkdir -p "$BRANCHES_DIR"
if [[ ! -f "$REGISTRY" ]]; then
  echo '{"branches":{},"next_slot":1}' > "$REGISTRY"
fi

# Check if branch already exists
if jq -e ".branches[\"${BRANCH}\"]" "$REGISTRY" >/dev/null 2>&1; then
  echo "Error: branch '${BRANCH}' already exists in registry"
  exit 1
fi

SLOT=$(jq -r '.next_slot' "$REGISTRY")
echo "[1/10] Allocated slot: ${SLOT}"

# ── 2. Create git worktree ──
echo "[2/10] Creating git worktree..."
cd "$REPO_ROOT"
git worktree add "$BRANCH_DIR" "origin/${BRANCH}"
echo "  Worktree: ${BRANCH_DIR}"

# ── 3. Create PostgreSQL database ──
echo "[3/10] Creating database: orcha_${BRANCH_SAFE}..."
if ! pg_isready -h localhost -p 5432 -q; then
  echo "Error: PostgreSQL is not running. Start it with: cd orcha && docker compose up -d"
  exit 1
fi
createdb -h localhost -U postgres "orcha_${BRANCH_SAFE}" 2>/dev/null || echo "  Database already exists"

# ── 4. Start LocalStack container ──
APP_PORT=$((18000 + SLOT))
ADMIN_PORT=$((17000 + SLOT))
LINK_PORT=$((19000 + SLOT))
LS_PORT=$((14000 + SLOT))

echo "[4/10] Starting LocalStack container: localstack-${BRANCH} (port ${LS_PORT})..."
docker run -d \
  --name "localstack-${BRANCH}" \
  -p "${LS_PORT}:4566" \
  -e PERSISTENCE=1 \
  -e "SERVICES=s3,sqs,secretsmanager,ssm" \
  -e DEFAULT_REGION=eu-central-1 \
  -v "localstack-${BRANCH}:/var/lib/localstack" \
  localstack/localstack:latest

# Wait for LocalStack to be healthy
echo "  Waiting for LocalStack..."
for i in $(seq 1 30); do
  if curl -sf "http://localhost:${LS_PORT}/_localstack/health" >/dev/null 2>&1; then
    break
  fi
  sleep 1
done

# ── 5. Generate .env file ──
echo "[5/10] Generating .env file..."
cat > "${BRANCH_DIR}/.env" <<EOF
ENVIRON_NAME=local-dev

# Ports
ORCHA_APP_PORT=${APP_PORT}
ORCHA_ADMIN_PORT=${ADMIN_PORT}
ORCHA_LINK_PORT=${LINK_PORT}

# External URLs
ORCHA_APP_BASE_URL=https://app-${BRANCH}.tdev.getorcha.com
ORCHA_LINK_BASE_URL=https://link-${BRANCH}.tdev.getorcha.com

# Internal endpoints
ORCHA_AWS_ENDPOINT=http://localhost:${LS_PORT}
ORCHA_DB_CREDENTIALS='{"host":"localhost","port":5432,"dbname":"orcha_${BRANCH_SAFE}","username":"postgres"}'
EOF


# ── 6. Init LocalStack resources ──
echo "[6/10] Initializing LocalStack resources..."
cd "${BRANCH_DIR}/orcha"
set -a; source "${BRANCH_DIR}/.env"; set +a
bb dev:init-aws

# ── 7. Build uberjar ──
echo "[7/10] Building uberjar..."
clojure -X:build

# ── 8. Run DB migrations ──
echo "[8/10] Running database migrations..."
bb migrate migrate

# ── 9. Create NPM proxy hosts ──
echo "[9/10] Creating NPM proxy hosts..."
"${SCRIPT_DIR}/npm-setup.sh" --branch "$BRANCH" --slot "$SLOT"

# ── 10. Update registry and enable systemd service ──
echo "[10/10] Enabling systemd service..."
jq ".branches[\"${BRANCH}\"] = {\"slot\": ${SLOT}, \"created\": \"$(date +%Y-%m-%d)\"} | .next_slot = $((SLOT + 1))" \
  "$REGISTRY" > "${REGISTRY}.tmp" && mv "${REGISTRY}.tmp" "$REGISTRY"

sudo systemctl enable --now "orcha@${BRANCH}"

echo ""
echo "=== Branch '${BRANCH}' is live ==="
echo "  App:   https://app-${BRANCH}.tdev.getorcha.com"
echo "  Admin: https://admin-${BRANCH}.tdev.getorcha.com"
echo "  Link:  https://link-${BRANCH}.tdev.getorcha.com"
echo "  Ports: app=${APP_PORT} admin=${ADMIN_PORT} link=${LINK_PORT} localstack=${LS_PORT}"
echo "  DB:    orcha_${BRANCH_SAFE}"
echo "  Logs:  journalctl -u orcha@${BRANCH} -f"

Step 2: Make executable

Run: chmod +x spikes/tolaria/scripts/branch-create.sh

Step 3: Syntax check

Run: bash -n spikes/tolaria/scripts/branch-create.sh Expected: no syntax errors

Step 4: Commit

git add spikes/tolaria/scripts/branch-create.sh
git commit -m "feat(tolaria): add branch provisioning script"

Task 5: Create branch-destroy.sh

Files:

Create: spikes/tolaria/scripts/branch-destroy.sh
Step 1: Write branch-destroy.sh

#!/usr/bin/env bash
set -euo pipefail

# Tear down an orcha branch environment on Tolaria.
# Usage: branch-destroy.sh <branch-name>

if [[ $# -lt 1 ]]; then
  echo "Usage: branch-destroy.sh <branch-name>"
  exit 1
fi

BRANCH="$1"
BRANCH_SAFE="${BRANCH//-/_}"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
BRANCHES_DIR="$HOME/orcha-branches"
BRANCH_DIR="${BRANCHES_DIR}/${BRANCH}"
REGISTRY="${BRANCHES_DIR}/registry.json"

echo "=== Destroying branch environment: ${BRANCH} ==="

# ── 1. Stop and disable systemd service ──
echo "[1/6] Stopping systemd service..."
sudo systemctl disable --now "orcha@${BRANCH}" 2>/dev/null || true

# ── 2. Remove NPM proxy hosts ──
echo "[2/6] Removing NPM proxy hosts..."
"${SCRIPT_DIR}/npm-setup.sh" --branch "$BRANCH" --teardown

# ── 3. Remove LocalStack container and volume ──
echo "[3/6] Removing LocalStack container..."
docker rm -f "localstack-${BRANCH}" 2>/dev/null || true
docker volume rm "localstack-${BRANCH}" 2>/dev/null || true

# ── 4. Drop PostgreSQL database ──
echo "[4/6] Dropping database: orcha_${BRANCH_SAFE}..."
dropdb -h localhost -U postgres "orcha_${BRANCH_SAFE}" 2>/dev/null || echo "  Database not found (skipped)"

# ── 5. Remove git worktree ──
echo "[5/6] Removing git worktree..."
cd "$REPO_ROOT"
git worktree remove "$BRANCH_DIR" --force 2>/dev/null || rm -rf "$BRANCH_DIR"

# ── 6. Free slot in registry ──
echo "[6/6] Updating registry..."
if [[ -f "$REGISTRY" ]]; then
  jq "del(.branches[\"${BRANCH}\"])" "$REGISTRY" > "${REGISTRY}.tmp" && mv "${REGISTRY}.tmp" "$REGISTRY"
fi

echo ""
echo "=== Branch '${BRANCH}' destroyed ==="

Step 2: Make executable

Run: chmod +x spikes/tolaria/scripts/branch-destroy.sh

Step 3: Syntax check

Run: bash -n spikes/tolaria/scripts/branch-destroy.sh Expected: no syntax errors

Step 4: Commit

git add spikes/tolaria/scripts/branch-destroy.sh
git commit -m "feat(tolaria): add branch teardown script"

Task 6: Create branch-update.sh

Files:

Create: spikes/tolaria/scripts/branch-update.sh
Step 1: Write branch-update.sh

#!/usr/bin/env bash
set -euo pipefail

# Update an orcha branch: pull latest, rebuild, restart.
# Usage: branch-update.sh <branch-name>

if [[ $# -lt 1 ]]; then
  echo "Usage: branch-update.sh <branch-name>"
  exit 1
fi

BRANCH="$1"
BRANCHES_DIR="$HOME/orcha-branches"
BRANCH_DIR="${BRANCHES_DIR}/${BRANCH}"

if [[ ! -d "$BRANCH_DIR" ]]; then
  echo "Error: branch directory not found: ${BRANCH_DIR}"
  exit 1
fi

echo "=== Updating branch: ${BRANCH} ==="

# ── 1. Pull latest ──
echo "[1/5] Pulling latest..."
cd "$BRANCH_DIR"
git pull

# ── 2. Build uberjar ──
echo "[2/5] Building uberjar..."
cd orcha
clojure -X:build

# ── 3. Init AWS resources (idempotent) ──
echo "[3/5] Updating LocalStack resources..."
set -a; source "${BRANCH_DIR}/.env"; set +a
bb dev:init-aws

# ── 4. Run migrations ──
echo "[4/5] Running database migrations..."
bb migrate migrate

# ── 5. Restart service ──
echo "[5/5] Restarting service..."
sudo systemctl restart "orcha@${BRANCH}"

echo ""
echo "=== Branch '${BRANCH}' updated ==="
echo "  Status: $(systemctl is-active orcha@${BRANCH})"
echo "  Logs:   journalctl -u orcha@${BRANCH} -f"

Step 2: Make executable

Run: chmod +x spikes/tolaria/scripts/branch-update.sh

Step 3: Syntax check

Run: bash -n spikes/tolaria/scripts/branch-update.sh Expected: no syntax errors

Step 4: Commit

git add spikes/tolaria/scripts/branch-update.sh
git commit -m "feat(tolaria): add branch update script"

Task 7: End-to-end test — create, verify, destroy a branch

This uses a real branch to test the full lifecycle. Create a throwaway branch for testing.

Step 1: Create a test branch on the remote

cd ~/orcha
git push origin tolaria:test-tolaria-e2e

Step 2: Provision the branch

Run: spikes/tolaria/scripts/branch-create.sh test-tolaria-e2e Expected: all 10 steps complete, URLs printed

Step 3: Verify systemd service is running

Run: systemctl is-active orcha@test-tolaria-e2e Expected: active

Step 4: Verify app is accessible

Open https://app-test-tolaria-e2e.tdev.getorcha.com in browser. Expected: orcha app loads (with basic auth)

Step 5: Verify the registry

Run: cat ~/orcha-branches/registry.json | jq . Expected: test-tolaria-e2e entry with slot 1

Step 6: Test branch-update

Run: spikes/tolaria/scripts/branch-update.sh test-tolaria-e2e Expected: pulls, rebuilds, restarts successfully

Step 7: Destroy the branch

Run: spikes/tolaria/scripts/branch-destroy.sh test-tolaria-e2e Expected: all 6 steps complete, resources cleaned up

Step 8: Verify cleanup

systemctl is-active orcha@test-tolaria-e2e 2>&1  # should be "inactive"
docker ps -a --filter name=localstack-test-tolaria-e2e --format '{{.Names}}'  # should be empty
psql -h localhost -U postgres -lqt | grep test_tolaria  # should be empty
ls ~/orcha-branches/test-tolaria-e2e 2>&1  # should not exist

Step 9: Clean up remote branch

git push origin --delete test-tolaria-e2e

Step 10: Commit any fixes

git add spikes/tolaria/
git commit -m "fix(tolaria): adjustments from end-to-end branch lifecycle test"