Future Infrastructure Improvements

This document tracks infrastructure improvements to implement as the project scales and generates revenue.

Current Trade-offs (Accepted for MVP)

Single Instance / Deployment Downtime

• Current: min=max=1 ASG with AllAtOnce deployment
• Impact: ~5-10 minute downtime during deployments and instance failures
• Acceptable for: MVP phase, low traffic, no SLA commitments

Public Subnet for EC2

See Risk Assessment below.

Completed Improvements

None yet

Planned Improvements

1. High Availability (When Revenue Justifies)

Current Cost: ~$30/month (1x t4g.medium) HA Cost: ~$60/month (2x t4g.medium) Additional Cost: +$30/month

Changes required:

• Set ASG min=2, max=2
• Change CodeDeploy to CodeDeployDefault.OneAtATime
• Consider RDS Multi-AZ (+$60/month for db.t4g.medium)

Total HA upgrade: +$90/month

2. VPC Endpoints

Priority: Medium (security + minor latency improvement)

Recommended first step: Add S3 Gateway Endpoint (free) immediately.

Full implementation: ~$122/month for all interface endpoints

Note: Data processing adds $0.01/GB through interface endpoints, but this is typically less than NAT Gateway costs for the same traffic.

3. WAF (Web Application Firewall)

Priority: Medium (once handling sensitive customer data)

Minimum WAF setup: ~$11/month Recommended setup (4-5 rule groups): ~$15/month

Consider adding when:

• Processing customer documents at scale
• Handling PII or financial data
• Compliance requirements emerge

4. ECR Retention Policy

Current: 5 images retained Consideration: Increase to 10-15 for better rollback capability

Cost impact:

• ECR storage: $0.10/GB/month
• Typical image size: 500MB-1GB
• 5 images: ~$0.50/month
• 15 images: ~$1.50/month
• Delta: +$1.00/month

Recommendation: Increase to 10 images. Negligible cost, better rollback safety.

5. NAT Gateway

Current: No NAT Gateway (EC2 in public subnet) Cost if added: ~$38/month + $0.052/GB processed

See Risk Assessment for why this is acceptable for now.

Public Subnet Risk Assessment

Current Architecture

EC2 instance in public subnet with:

• Public IP address
• Security group allowing only ALB on port 8888
• No SSH port (22) exposed (using SSM Session Manager)
• Outbound 443 to internet for AWS APIs

Actual Risks

What Public Subnet Actually Exposes

1. Network path exists from internet to instance (but SG blocks it)
2. Public IP is visible (but no services listening except via ALB)
3. Outbound traffic goes directly to internet (vs through NAT)

What It Does NOT Expose

• No SSH port
• No direct HTTP/HTTPS (only through ALB)
• No database access (RDS in private subnet)
• Secrets are in Secrets Manager, not on disk

Comparison: Public Subnet vs NAT Gateway

Recommendation

Keep public subnet for MVP. The security group is the primary defense in both architectures. Move to private subnet + NAT when:

• Pursuing compliance certifications
• Processing highly sensitive data
• Revenue exceeds infrastructure cost concerns

Migration Testing Strategy

Current Plan

Run migration tests only on schema-changing merges, not every deployment.

Detection

Add a CodeBuild step that:

1. Checks if any resources/migrations/*.sql files changed
2. If yes, trigger migration test pipeline
3. If no, skip directly to deploy

Cost Impact

• Migration test creates temporary RDS instance (~10 minutes)
• db.t4g.medium: $0.073/hour → ~$0.012 per test
• Snapshot restore: Free (no additional charge)
• Estimated: $0.50-2.00/month depending on merge frequency

Stack Consolidation Proposal

Current: 9 Stacks

NetworkStack → StorageStack → QueueStack → EcrStack → DatabaseStack
                    ↓
               DnsStack → ComputeStack → MonitoringStack → CicdStack

Proposed: 4 Stacks

1. FoundationStack (replaces Network + Storage + Queue + ECR)

• VPC, subnets, security groups
• S3 buckets
• SQS queues
• ECR repository
• SSM parameters

2. DataStack (replaces Database)

• RDS PostgreSQL
• Secrets Manager secret

3. ComputeStack (replaces DNS + Compute)

• ACM certificate
• Route53 record
• ALB
• ASG + Launch Template
• Target group

4. OpsStack (replaces Monitoring + CICD)

• CloudWatch log groups and alarms
• SNS topic
• Cost anomaly detection
• CodePipeline + CodeBuild + CodeDeploy

Benefits

• Fewer cross-stack references
• Simpler rollback scenarios
• Faster deployments (fewer stacks to update)
• Clearer mental model

Migration Path

1. Create new consolidated stacks
2. Import existing resources (CDK resource importing)
3. Delete old stacks

Effort: ~4-6 hours Risk: Medium (resource importing can be tricky) Recommendation: Do this before first production deployment, not after

Cost Summary

Automatic Database Credential Rotation

Current: Manual rotation only. Database credentials are stored in Secrets Manager (/v1-orcha/db-credentials). After manual rotation, restart the application to pick up new credentials.

Future: Enable automatic rotation with application restart.

Implementation

1. Enable RDS Secrets Manager rotation on the secret
2. Add EventBridge rule to catch SecretsManager Secret Rotation Successful event
3. Trigger application restart via one of:
   • Lambda that calls autoscaling:StartInstanceRefresh
   • Lambda that triggers CodeDeploy deployment
   • Lambda that terminates instance (ASG launches new one)

Alternative: Split Credentials

If you need rotation without restart:

1. Change to separate DB_HOST env var + v1-orcha/db-credentials secret (username/password only)
2. Use AWS JDBC Driver which has built-in Secrets Manager integration and auto-refresh
3. Or implement credential refresh logic in the app (catch auth failures, refetch, rebuild pool)

Cost

• Secrets Manager rotation: included (no extra cost)
• EventBridge rule: free
• Lambda invocations: negligible

Per-Tenant Resource Scaling

When a large tenant requires dedicated resources to avoid noisy-neighbor issues:

Resources to Create Per Tenant

1. SQS Queues:

   • v1-orcha-{tenant-slug}-ingest + v1-orcha-{tenant-slug}-ingest-dlq
   • v1-orcha-{tenant-slug}-email-acquire + v1-orcha-{tenant-slug}-email-acquire-dlq

2. S3 Bucket:

   • v1-orcha-{tenant-slug}-storage-{account_id}

3. Dedicated Worker Instance(s):

   • Configured to consume from tenant-specific queues
   • Can have higher memory/CPU allocation for large document volumes

4. Optional: Dedicated API Keys:

   • Per-tenant Anthropic/Google API keys stored in SSM or Secrets Manager
   • Provides rate limit isolation and cost attribution

Implementation Steps

1. Create tenant-specific AWS resources (queues, bucket)
2. Store resource configuration in database (tenant table or config table):
   • Queue names/URLs
   • S3 bucket name
   • API key parameter paths (if dedicated)
3. Update ERP to route work to tenant-specific queue based on tenant config
4. Deploy dedicated worker instance(s) with tenant-specific config
5. Monitor and scale worker instances independently

Cost Considerations

Recommendation: Only create dedicated resources when tenant volume justifies the operational overhead. Start with dedicated workers consuming from global queues, escalate to full isolation only if needed.

Revision History