Delta moves to Cape Town — and learns to scale.

If you hold four ideas in your head, almost everything else follows. The rest of this doc is detail hanging off these.

Today Delta is a single c7g.large in Ireland, managed by Laravel Forge, running every role — web, queues, cron, WebSockets, and Redis — on the one machine.

It has served us well, but it's out of room:

• ~180 ms latency tax. Users are in South Africa, the server is in Ireland. For an Inertia app where every navigation is a full server round-trip, that's seconds of dead time per session.
• Single point of failure. A runaway job, memory spike, or Redis hiccup takes down everything at once.
• No isolation. Horizon workers fight PHP-FPM for CPU and RAM — a heavy import or export degrades page loads for everyone.
• No way to scale. Peak school hours (08:00–15:00 SAST) push the one box near capacity, with no headroom to absorb a spike.

Cloudflare → a load balancer → a fleet of stateless web boxes that grows and shrinks, plus one worker for the run-once jobs. State is pushed out to managed services (Redis, MySQL, EFS). All in Cape Town, ~10 ms from users.

The request path: a browser hits a tenant domain → Cloudflare → the ALB. Normal traffic goes to whichever web instance is free; anything under /app/* (Reverb WebSockets) is routed to the worker. Web boxes and the worker both live in private subnets with no public IP — the only way in from the internet is through the ALB, and the only way out is through the NAT Gateway.

Same application, very different operational shape. The amber column is what we're retiring; teal is where we're going.

You push to main. That's the whole action. GitHub Actions builds a release, ships it to the worker first (so migrations run once), then to the web fleet, verifies the load balancer sees everyone healthy, and rolls back automatically if it doesn't. About 3–4 minutes end to end.

A few things worth knowing:

• Worker-first ordering is load-bearing. The worker runs migrations before any web box switches to the new code, so web instances never run against a schema that doesn't exist yet. (This is why the manual deploy scripts warn against targeting all instances at once.)
• Tests are the PR gate, not the deploy. The full Pest matrix runs on every PR into main (on Blacksmith runners). The deploy pipeline deliberately has no test step, to stay fast. The accepted exposure: a direct push to main deploys untested — branch protection is the fix and is tracked separately.
• Two pointers: latest and stable. New instances boot from latest. stable is only advanced after the final health check passes, so it's always a known-good release to roll back to.
• Rollback is automatic. If the health gate fails, the pipeline restores stable → latest and redeploys every running box in rollback mode. Forward-only migrations are not reversed — recovery is code-level, which is why we enforce expand-and-contract migrations.

Depends what you're changing. There are four distinct paths, and it's worth being clear on which is which — they have very different blast radii.

Changing app config

The .env lives in Secrets Manager (/delta/production/env), not on any server. Pull it, edit it, run the same preflight the boxes run (APP_ENV/APP_KEY present, ≤64 KB), push a new secret version, then deploy so instances re-fetch it. A failed fetch never overwrites the live file.

Changing infrastructure

Edit the Terraform in infrastructure/terraform/, run terraform plan, get a review, apply. How careful you need to be scales with the change:

Staging is an ephemeral mirror — the same Terraform in a second workspace (10.1.0.0/16, smaller instances, its own RDS restored from a PII-sanitised seed snapshot). Spin it up (staging-up.sh, ~25–35 min), test, tear it down (staging-down.sh) so idle cost is near-zero. A one-time full rehearsal on staging is a gate before the production cutover.

Getting a shell

No SSH keys, no public IPs. Everything is AWS Session Manager: aws ssm start-session --target <instance-id>. You'll almost always want the worker — that's where Horizon, the scheduler, and artisan commands live (cd /var/www/delta/current && sudo -u deploy php artisan …).

Scaling for a known event

Big onboarding tomorrow? Raise the floor ahead of time with aws autoscaling set-desired-capacity … --desired-capacity 4. It drifts back down automatically (schedules re-assert the floor at 07:00/17:00 SAST; CPU-based scaling handles surprises in between). The ASG max of 5 is the real ceiling.

Two layers. Nightwatch is the first stop for anything app-level; CloudWatch owns infra metrics, logs, and the alarms that email you. Alerts arrive by email from CloudWatch via SNS.

What's alarmed

Triage order for common symptoms lives in §07 and the ops docs. Rule of thumb: 5xx or user-reported errors → CloudWatch nginx/php-fpm logs + Nightwatch; jobs stuck → SSM into the worker, php artisan horizon:status.

Mostly, the system heals itself — that's the point of cattle. The exceptions are the stateful and single-instance pieces, and those are worth knowing cold.

Materially more, on almost every axis — with one deliberate, contained exception (a publicly-reachable database endpoint for BI tools).

• Origin lockdown. The ALB only accepts traffic from Cloudflare's published edge ranges (plus admin/VPN CIDRs). Anyone who finds the ALB DNS name via Certificate Transparency logs still can't bypass Cloudflare's DDoS/WAF/cache. A weekly CI job fails loudly if Cloudflare's IP list drifts.
• No SSH. Access is SSM Session Manager only; instances have no public IPs and IMDSv2 is required. No key material to leak or rotate.
• Private compute. Web and worker sit in private subnets; the only inbound path is the ALB, enforced by security-group-to-security-group rules (not IP ranges).
• Secrets stay secret. The .env is in Secrets Manager, never committed, never baked into an image. Deploys authenticate via GitHub OIDC — no static AWS keys, and trust is scoped to the main branch.
• Encryption everywhere. At rest on EBS, EFS, RDS, Redis, and S3; in transit to Redis (TLS + auth token).

Roughly $130–175/month excluding RDS and S3, up from ~$60. The variability is the web fleet flexing with the school day. The hard ceiling is the ASG max of 5 instances — the budget alarm informs, it can't stop spending.

What the ~$70–110/mo increase buys: the 10–15× latency win, a self-healing auto-scaling web tier, an isolated worker so jobs can't degrade page loads, managed Redis with real headroom, zero-downtime deploys with instant rollback, and the whole thing as reviewable, reproducible code.

Three independent layers for the database and shared files — and, importantly, restore drills are a documented, rehearsed step, not a hope.

• EFS is protected in prod with Terraform prevent_destroy — removing that guard is itself a reviewed code change.
• Restore is proven, not assumed. Ops docs carry step-by-step drills for both the DB dump and the EFS recovery point — restored into an isolated throwaway target, with row-count checks recorded. Proving a restore is a gate before cutover.
• What we don't back up ourselves: user files live in S3/EFS with their own durability, so the logical backup is deliberately database-only — nightly zips of storage/ added size for no recovery value.

A distributed system buys resilience and scale at the price of moving parts. These are the real trade-offs and the risks we've knowingly accepted — the parts a colleague should push on.

Things that genuinely got harder

• Maintenance mode is per-instance. php artisan down writes a marker into local storage/framework, so it only downs the box you ran it on. Fleet-wide maintenance needs an SSM fan-out (documented) or a Cloudflare/ALB rule.
• Debugging spans machines. No single box holds the whole story any more. Mitigated by centralising app telemetry in Nightwatch — but it's a mindset shift from "SSH into the box."
• The deploy pipeline is more than a git pull. More capable (health gates, rollback, worker-first) but more machinery to understand when something in it misbehaves.
• No more Forge dashboard. Ad-hoc server management is now SSM + Terraform + AWS console instead of a friendly UI. More power, steeper day-to-day.
• Shared storage has NFS latency. storage/app on EFS is slower than local disk — which is exactly why compiled views and framework caches were kept on local disk, off the request hot path.

Accepted risks (deliberate, with upgrade paths)

A canary phase against an isolated copy of the data, then a short Saturday maintenance window built around promoting a cross-region database replica. Rollback is free right up until the database point-of-no-return.

Canary — prove it in isolation (days 1–3)

Cape Town connects only to a dedicated delta-canary RDS restored from an Ireland snapshot — never the live database. A single test tenant is pointed at the new ALB via a DNS override. External integrations get canary-safe overrides (e.g. a canary_ Algolia prefix, mail to log) so week-old snapshot data can't write to production systems. The phases:

1. Smoke test — infra health, ALB, EFS, worker services, WebSocket ingress.
2. Single-tenant canary — one real tenant on the new stack + 24 h soak.
3. Load test — verify scale-out, scale-in, and scheduled scaling actually work.
4. Deploy pipeline — a real deploy, a real rollback, deploy under traffic, scale-out during a deploy.

Cutover — the Saturday window (phase 5)

Days ahead, a cross-region read replica of the Ireland RDS is created in Cape Town and left syncing. The window itself is short:

1. Old site → maintenance; stop the scheduler; let Horizon drain; stop Horizon (the drain gate — no in-flight jobs lost).
2. Confirm replica lag is zero.
3. Promote the Cape Town replica — minutes, and the point of no return.
4. Update the .env secret to point at the promoted database; deploy so everyone picks it up.
5. Switch the wildcard DNS records to the ALB; verify.

After (phases 6–7)

The old Ireland box stays as a silenced cold standby for a week — in maintenance mode with Horizon and the scheduler stopped, so it can't double-send notifications or double-sync Zoho — then it's decommissioned.

Scope was held tight to de-risk the migration. These are known, deferred, and cheap to revisit when the need is real.

• Multi-AZ NAT egress — a second NAT Gateway + per-AZ routing. Revisit when the availability gain justifies the monthly cost.
• Multi-AZ RDS — automatic DB failover; doubles RDS cost.
• Moving S3 buckets to af-south-1 — app buckets stay in Ireland initially; user downloads go browser-direct from S3, so impact is mostly on background jobs.
• CloudFront — may be redundant given Cloudflare already does edge caching. Evaluate before adding.
• PostgreSQL — planned separately to avoid compounding migration risk.
• Containers (ECS/Fargate) — only if sustained scale outgrows ~3 instances. The ASG approach is simpler and sufficient now.

The handful of commands that cover 90% of day-to-day operations. All in af-south-1.

# Just push to main.
git push origin main
# Watch it in the repo's Actions tab.

aws ssm start-session \
  --target <instance-id> \
  --region af-south-1

cd /var/www/delta/current
sudo -u deploy php artisan horizon:status

aws autoscaling set-desired-capacity \
  --auto-scaling-group-name delta-web-asg \
  --desired-capacity 4 --region af-south-1

aws logs tail /delta/web/nginx \
  --follow --region af-south-1
# app-level: Nightwatch dashboard

# Don't debug it. Terminate it —
# the ASG boots a healthy replacement.