Real-Time Trading Data Platform — AWS Architecture Portfolio

01 The big picture

A single Kinesis Data Stream is the fan-out point. One branch feeds a Lambda consumer for immediate processing; the other branch feeds Firehose for buffered delivery into a data lake that Glue and Athena turn into SQL-queryable tables.

flowchart LR
  subgraph net["VPC — private subnets, no NAT/IGW route at runtime"]
    ecr["Amazon ECR
producer image"]
    ecs["ECS Fargate
producer (simulated trades)"]
    ecr -->|"pull via VPC endpoint"| ecs
  end

  ecs -->|"put records"| kds["Kinesis
Data Stream"]

  kds -->|"real-time trigger"| lam["Lambda
consumer"]
  kds -->|"buffered"| fh["Kinesis
Firehose"]

  lam -->|"active positions"| ddb[("DynamoDB")]
  lam -->|"completed trades"| s3done[("S3
completed-trades/")]

  fh --> s3raw[("S3
raw-data/")]
  s3raw --> crawler["Glue Crawler"]
  crawler --> catalog["Glue Data Catalog"]
  catalog --> athena["Athena
SQL analytics"]
  athena --> s3res[("S3
Athena results")]

  classDef rt fill:#12351f,stroke:#3fb950,color:#e6edf3;
  classDef batch fill:#22303f,stroke:#4493f8,color:#e6edf3;
  class lam,ddb,s3done rt;
  class fh,s3raw,crawler,catalog,athena,s3res batch;

Green = real-time path (event-driven, low latency). Blue = batch analytics path (buffered, cost-optimized). The Glue crawler actually catalogs both the raw-data/ and completed-trades/ prefixes. The producer image is promoted from Docker Hub into ECR at deploy time, then pulled from ECR over a VPC endpoint — no public internet at runtime.

02 The two data paths, side by side

The same stream serves two very different consumers. Understanding why is the core of this design.

flowchart TB
  start(["Trade event on Kinesis"])

  subgraph RT["Real-time path — low latency"]
    direction TB
    r1["Lambda consumer"] --> r2["Detect buy/sell signal"]
    r2 --> r3[("DynamoDB
open positions")]
    r2 --> r4[("S3 completed-trades/")]
  end

  subgraph BATCH["Batch path — cost-optimized"]
    direction TB
    b1["Firehose buffers batches"] --> b2[("S3 raw-data/")]
    b2 --> b3["Glue Crawler discovers schema"]
    b3 --> b4["Glue Data Catalog"]
    b4 --> b5["Athena SQL, no ETL jobs"]
  end

  start --> r1
  start --> b1

  classDef rt fill:#12351f,stroke:#3fb950,color:#e6edf3;
  classDef batch fill:#22303f,stroke:#4493f8,color:#e6edf3;
  class r1,r2,r3,r4 rt;
  class b1,b2,b3,b4,b5 batch;

Real-time path keeps current state (positions) and archives finished trades. Batch path stores everything raw and lets schema-on-read analytics run later — no ETL jobs to maintain. The buy/sell logic in the consumer is a deliberate demonstration skeleton (a simple price threshold), not a real trading strategy — the project's focus is the event-driven pipeline, not the algorithm.

	Real-time path	Batch analytics path
Consumer	Lambda (Kinesis trigger)	Firehose
Latency	Near real-time	Buffered (seconds–minutes)
Stores	DynamoDB + S3 completed-trades/	S3 raw-data/
Queried by	Application reads	Athena SQL
Optimized for	Speed	Cost & flexibility

03 Staged, dependency-ordered deployment

The infrastructure is split into a bootstrap stage plus five ordered stacks. Each stack consumes the outputs of earlier ones, so deployment (and teardown) follows a clean dependency order with no cycles.

flowchart LR
  s0["0-bootstrap
OIDC + CI IAM role"]
  s1["foundation
VPC · subnets · endpoints"]
  s2["data-streaming
Kinesis · Firehose · S3 lake"]
  s3["producers
ECS Fargate · ECR"]
  s4["consumers
Lambda · DynamoDB"]
  s5["analytics
Glue · Athena"]

  s0 --> s1 --> s2 --> s3
  s2 --> s4
  s2 --> s5
  s4 --> s5

  classDef n fill:#1c2330,stroke:#ff9900,color:#e6edf3;
  class s0,s1,s2,s3,s4,s5 n;

Reusable modules (foundation, data-streaming, producers, consumers, analytics, s3) are composed by the stages. Destroy runs in reverse dependency order for a clean teardown with no orphaned resources.

04 Network isolation & a real-world egress tradeoff

ECS tasks run in private subnets with no NAT gateway and no internet route. AWS services are reached through VPC endpoints. There is one honest, well-documented exception worth calling out.

Isolation model. Public S3 access is fully blocked, IAM roles follow least privilege, and there are no static credentials. ECR, S3, and CloudWatch are reached via VPC endpoints rather than the public internet.

The ECS egress tradeoff. ECR image-layer delivery uses AWS-managed backing infrastructure whose IPs are not covered by a stable CIDR or endpoint security group. Restricting egress to the endpoint SG causes CannotPullContainerError. So the task SG allows outbound TCP/443 to 0.0.0.0/0 — but this grants no internet access, because the tasks sit in private subnets with no NAT or IGW route. Isolation is enforced at the route-table level; IAM enforces service-level access.

Design decisions

Event-driven, fully serverless after ingestion (Lambda + managed services).
Glue Data Catalog used as a metadata layer over S3 — schema-on-read, no ETL jobs.
Runtime internet dependency removed via ECR + VPC endpoints.
Immutable infrastructure with clean, cost-safe teardown.