Every request-path service runs as paired A / B replicas. The reverse proxy fronts each pair with a round-robin upstream pool, an upstream health probe, and passive failover. The sibling carries traffic while its twin recreates.
The rolling-deploy script typechecks the SPA on the workstation first, SSHes to the box, and rolls one replica at a time, gating on the health probe before the next replica flips. Code, environment, or credential changes deploy with no user-visible 502 window.
The same topology carries to ECS / Fargate, EKS, or Fly.io by swapping the orchestrator. Service code, container images, and the reverse-proxy upstream contract are unchanged.
Eight phases, one audit row
Every deploy — dev or prod — runs the same pipeline. Halt-on-any-failure. A trap-on-exit writes the terminal outcome to the deploy-history table regardless of which phase failed, so the audit trail is complete even when the deploy aborts mid-flight.
- Build — one image tagged with the new commit SHA, reused by every later phase.
- Migrate pre-roll — apply the additive, expand, switch, and data migrations. Old code keeps running against the new schema.
- Record-start — write a deploy-history row in the in-progress state.
- Roll A-side — recreate the A-side of every replica pair, gate on the health probe, then a five-minute soak, then an SLO-verify step that reads recent server-errors and exits non-zero on breach.
- Roll B-side — same shape. Second five-minute soak. Second SLO-verify gate. The reverse proxy reloads last, so its routing config sees healthy upstreams from moment zero.
- Migrate contract — apply the contract-phase migrations. Destructive SQL runs now that the old code is gone.
- Smoke — a signed bearer derived at runtime from the platform master key posts to a deploy-smoke endpoint. The endpoint runs three server-side canaries as proxies for auth, entity read, and the agent write path.
- Housekeeping and record-finish — dangling images pruned, build cache capped, terminal outcome recorded.
Zero new environment variables for the smoke surface. The master key is the credential; the bearer derives at runtime; a tight clock-skew window makes it effectively single-use.
Expand, switch, contract
Every migration file declares its phase in a header — additive, expand, switch, contract, or data — and the discipline is enforced in two independent places.
A merge-time gate rejects any pull request that pairs a destructive verb with any phase other than contract. The runtime migration runner refuses to apply a file whose header is missing or unrecognised, so the discipline holds even if the merge-time gate is somehow bypassed.
The two buckets the pipeline runs — pre-roll for additive, expand, switch, data; post-roll for contract only — guarantee the previous-version application can serve traffic against the new schema for one full release cycle. That is what makes rollback safe.
Backward-compatible changes ship in one deploy. Destructive changes ship in two, with a gap: the first stops using the old shape; the second drops it. A column rename becomes three migrations across two deploys — expand, switch, contract. The naïve one-shot rename fails the merge-time gate before it lands.
One-shot migrate container, content-hash drift detection
The migrator is a one-shot container the pipeline invokes with a detached I/O contract that survives being run under an SSH heredoc without silently consuming its stdin.
The same migration runner powers both this CLI invocation and the in-app migration surface at /platform/operations/db-admin. Content-hash recording, drift detection, and advisory-lock serialisation are identical across surfaces. A drifted file surfaces as a diff between the recorded hash and the on-disk file. One code path, one audit trail.
Every paid feature has a live probe
/platform/operations is the single page that tells an operator whether the platform is functional end-to-end. Container health is one row; the other rows exercise every paid feature. Six tiers: per-replica HA health across every paired service, host resources (root filesystem pressure, degraded at 80% used, down at 90%), both object-storage buckets probed with region rendered, agent runtime (LLM providers, embeddings, Whisper, workflow worker pulse, billing worker pulse, meeting-recorder retention sweeper heartbeat), Postgres backup recency, and the OpenTelemetry collector.
LLM probes are server-cached for 30 seconds so the Refresh button doesn’t hammer providers. A degraded probe raises a platform-admin banner on every /platform/* page, so an operator sees pressure the moment they open any admin surface. The pitch line is literal: every paid feature on the platform has a live reachability probe so silent vendor outages surface to operators before customers do.
Fail-loudly substrate
Every uncaught failure on either side of the wire becomes a forensic row in the same Postgres the application writes to. One global server-side exception handler writes to a server-errors table with full traceback, request id, exception type, method, path, status, tenant, and actor. Browser-tier failures land in a frontend-errors table from the standard client error events and the SPA root error boundary. Both surfaces poll their triage UIs every 30 seconds.
Routers hold no endpoint-level catch surface. A sweep removed every silent catch across the router layer; the “catch-Exception, return generic error” pattern is structurally impossible now. The error-boundary copy “An administrator has been notified” is literally true — by the time the recovery UI paints, the forensic row is in the database.
OpenTelemetry across every tier
Every service and the SPA emit through the OpenTelemetry standard using OTLP over HTTP. A W3C traceparent header is auto-injected on every same-origin API call, so one trace spans browser through every backend hop. The reverse proxy passes the OTLP export path through to the collector for same-origin export. OTLP is a public specification with multiple compatible backends — an open-standards property, vendor-neutral by construction.
The takeaway
Deploys are boring on purpose. Two invocations of the deploy script cover the 1% of destructive changes; one covers the other 99%. The audit trail is a table you can query. Rollback is one command against the last successful commit SHA, drilled quarterly under SOC 2 CC7.4. For a technical buyer, the story sits at the layer that matters: durable execution on Postgres, open standards at every boundary, and an operations posture that reads like infrastructure.