Project 1 · Reliability & Disaster Recovery

Multi-Region Disaster Recovery Platform

A production-style WordPress platform running on ECS Fargate, replicated across two AWS regions in a warm-standby disaster-recovery model. Read traffic fails over automatically at the CDN layer; full recovery (database promotion, compute scale-up) is operator-triggered and then executed automatically by Step Functions.

This page is the visual overview. Full source code, deploy guide, cost breakdown, and demo videos live on GitHub.

01 The big picture

Two fully isolated VPCs, one per region. us-east-1 is the active primary; ca-central-1 is the warm standby. Global services — Route 53, CloudFront, and the Step Functions recovery workflow — sit above both regions and steer traffic between them.

flowchart TB
  user([End user / browser])

  subgraph GLOBAL["Global services"]
    r53["Route 53
DNS + failover records"] cf["CloudFront
2 origin groups w/ auto-failover"] end user --> r53 --> cf subgraph PRIMARY["Primary region — us-east-1 (ACTIVE)"] direction TB albP["ALB (primary)"] ecsP["ECS Fargate
WordPress · 2 tasks"] rdsP["RDS MySQL
writer"] s3P["S3 media bucket
(primary)"] albP --> ecsP ecsP --> rdsP ecsP --> s3P end subgraph DR["DR region — ca-central-1 (WARM STANDBY)"] direction TB albD["ALB (DR)"] ecsD["ECS Fargate
WordPress · 0 tasks"] rdsD["RDS MySQL
read replica"] s3D["S3 media bucket
(DR)"] albD --> ecsD ecsD -.-> rdsD ecsD --> s3D end cf -->|"app origin group
primary → DR"| albP cf -.->|failover| albD cf -->|"media origin group
/wp-content/uploads/*
primary → DR"| s3P cf -.->|failover| s3D rdsP ==>|"async cross-region
replication"| rdsD s3P ==>|"cross-region
replication (CRR)"| s3D classDef active fill:#12351f,stroke:#3fb950,color:#e6edf3; classDef standby fill:#3a1d1d,stroke:#f85149,color:#e6edf3; class albP,ecsP,rdsP,s3P active; class albD,ecsD,rdsD,s3D standby;
Solid lines = normal traffic and continuous replication. Dashed lines = failover paths that activate only during an incident. Green nodes are active; red nodes are warm standby (note ECS DR runs 0 tasks until failover).

02 Why admin traffic bypasses the CDN

A subtle but important design decision: public/frontend traffic goes through CloudFront, but the WordPress admin dashboard is routed directly to the ALB via Route 53. CloudFront origin groups do not support the POST requests that WordPress login and content writes require, so sending admin traffic through the CDN would break the dashboard.

flowchart LR
  subgraph pub["Public traffic"]
    d1["rqays.com
www.rqays.com"] --> cf1["CloudFront"] --> a1["ALB
(primary → DR failover)"] end subgraph adm["Admin traffic"] d2["admin.rqays.com"] --> r53f["Route 53
failover routing"] --> a2["Primary ALB → DR ALB"] end classDef n fill:#1c2330,stroke:#ff9900,color:#e6edf3; class d1,cf1,a1,d2,r53f,a2 n;
GET-heavy public traffic benefits from CDN caching and edge failover. POST-heavy admin traffic takes the direct Route 53 → ALB path so writes and logins work reliably.

03 How a region failover actually runs

Read availability fails over automatically at the CloudFront origin-group layer (the origin groups fail over on HTTP 403, 404, 500, 502, 503, 504, plus connection timeouts and origin unreachability). But full recovery — promoting the database and bringing compute online — is deliberately operator-triggered, then executed automatically by a Step Functions workflow. This avoids flapping between regions on transient errors.

sequenceDiagram
  actor Op as Operator
  participant CF as CloudFront
  participant SFN as Step Functions
  participant RDS as RDS (DR replica)
  participant ECS as ECS (DR service)
  participant App as App health check

  Note over CF: Primary origin returns 403/404/5xx or times out
  CF->>CF: Auto-failover reads to DR ALB & DR S3
  Note over Op: Operator confirms incident, starts workflow
  Op->>SFN: Start DR failover execution
  SFN->>RDS: ReplicaFailoverHandler — promote replica
  SFN->>RDS: ValidateDBWritable — confirm writable
  RDS-->>SFN: DB is now a writer
  SFN->>ECS: ServiceRecoveryHandler — scale 0 → 2 tasks
  ECS-->>SFN: Tasks healthy, registered to DR target group
  SFN->>App: ValidateApplication — HTTP health check
  App-->>SFN: 200 OK
  SFN-->>Op: Failover complete (auditable execution log)
        
The four-step state machine: promote replica → validate writable → scale DR compute → validate application. Read traffic is already being served from DR before step 1 even begins.
LayerFailover behaviorTrigger
Application reads (ALB)Automatic via CloudFront origin groupHTTP 403/404/500/502/503/504, timeout, unreachable
Media reads (S3, /wp-content/uploads/*)Automatic via CloudFront S3 origin groupSame origin-failover codes
Database (RDS)Operator-triggered, then automatedStep Functions execution
Compute (ECS)Scaled 0 → 2 by the workflowStep Functions execution

04 How the Terraform is organized

The code is split into reusable modules and environment stacks (bootstrap, global, primary, dr, operations). Each region deploys independently, which prevents cross-region dependency cycles and keeps the two regions truly isolated. State is remote (S3) per stack.

flowchart TB
  boot["bootstrap
OIDC provider + GitHub Actions IAM role"] subgraph modules["Reusable modules"] m["vpc · sg · alb · ecs · rds · s3
acm · cdn_dns · endpoint · iam · lambda"] end subgraph global["environments/global"] g1["iam"]:::g g2["oac"]:::g g3["cdn_dns"]:::g end subgraph prim["environments/primary"] p1["network → rds → s3 → alb → ecs"]:::p p2["failover-alarms"]:::p end subgraph drenv["environments/dr"] d1["network → read_replica_rds → s3 → alb → ecs"]:::d end ops["operations/dr_orchestration
Step Functions + Lambda handlers"] boot --> global modules -.consumed by.-> global modules -.consumed by.-> prim modules -.consumed by.-> drenv prim --> ops drenv --> ops global --> ops classDef g fill:#22303f,stroke:#4493f8,color:#e6edf3; classDef p fill:#12351f,stroke:#3fb950,color:#e6edf3; classDef d fill:#3a1d1d,stroke:#f85149,color:#e6edf3;
Stack outputs flow between stacks (bootstrap → global/primary/dr → operations). Modules are the shared building blocks; environments compose them per region.
Secure CI/CD. GitHub Actions authenticates to AWS with OpenID Connect (OIDC) — no static access keys or long-lived secrets. The IAM trust policy is scoped to a single repository, and deploy/destroy are separate workflows.

05 Design decisions & tradeoffs

The most interview-relevant part of this project is that the tradeoffs are deliberate, not accidental.

Warm standby over active-active. Active-active would cut recovery time but introduces database consistency and conflict-resolution complexity. Warm standby keeps stateful components under control and cuts DR compute cost by 50–70% (DR ECS runs 0 tasks until failover).
Operator-triggered database failover. Auto-promoting the replica risks flipping regions on a transient blip. Requiring human confirmation trades a slightly higher RTO for predictability and auditability.
RPO > 0. Cross-region MySQL replication is asynchronous, so a failover may lose the most recent seconds of writes. Zero data loss would require synchronous or multi-master replication at significantly higher cost.
Region-scoped IAM over tag-based conditions for RDS-managed secrets. Tag-based IAM conditions failed non-deterministically outside us-east-1 because RDS-managed secret tags propagate with eventual consistency. The project scopes access by account + region + ARN pattern instead, favoring deterministic, region-safe behavior.

Well-Architected mapping

PillarImplementation
ReliabilityMulti-region, CDN auto-failover, RDS replica, orchestrated recovery
SecurityTLS everywhere, IAM roles (no keys), Secrets Manager, least-privilege SGs, private subnets
PerformanceCloudFront CDN, S3 media offload, Fargate
Cost optimizationWarm standby, VPC endpoints to avoid NAT charges
Operational excellenceFull IaC, OIDC CI/CD, auditable Step Functions runbook
← Back to all projects