Almost the entire SkyHub backend runs as six containers on a single ECS Fargate cluster, skyhub-prod-cluster, in eu-central-1 (account <aws-account-id>). There is no RDS instance, no ElastiCache, and no load balancer in the account — Postgres and Redis are ordinary containers on the same cluster, and all public ingress arrives through the WireGuard/nginx jumphost rather than an ALB (see VPC, WireGuard Jumphost & nginx Routing). Each service is a terraform-aws-module-style module under skyhub_terraform/modules/ that creates one aws_ecs_task_definition + one aws_ecs_service (desired count 1) and registers an A record in AWS Cloud Map under the private DNS namespace skyhub-prod.internal (modules/ecs/ecs.tf). Services find each other exclusively by these *.skyhub-prod.internal names — there are no hardcoded task IPs.
This page documents the task definitions: resources, architecture, ports, env wiring, and the API redeploy-loop alarms. For the cluster/VPC big picture see AWS Infrastructure Overview; for the nginx routing that fronts these ports see Networking & Jumphost; for the full gateway env-var catalog see Gateway Environment Variables.

The cluster

skyhub_terraform/modules/ecs/ecs.tf
module "ecs" {
  source             = "terraform-aws-modules/ecs/aws"
  name               = "${var.resources_tag}-cluster"   # skyhub-prod-cluster
  version            = "3.5.0"
  container_insights = true
  capacity_providers = ["FARGATE", "FARGATE_SPOT"]
  default_capacity_provider_strategy = [
    { capacity_provider = "FARGATE_SPOT", weight = 100 }
  ]
}

resource "aws_service_discovery_private_dns_namespace" "dns_service" {
  name = "${var.resources_tag}.internal"   # skyhub-prod.internal
  vpc  = var.vpc_id
}
  • Container Insights is on, which is what makes the ECS/ContainerInsights metrics (used by the API alarms below) available.
  • The cluster’s default strategy is 100% FARGATE_SPOT, but every service pins launch_type = "FARGATE", which overrides that default and places tasks on standard on-demand Fargate. To actually move a service onto Spot you must drop launch_type and add a capacity_provider_strategy block — changing only the cluster default has no effect.
  • There are 0 registered container instances — this is pure serverless Fargate.

Service inventory

The private IPs below are the live values captured from prod (skyhub-prod.internal Cloud Map zone); they change on every task replacement, so always resolve the DNS name, never the IP.
ECS serviceCloud Map DNS (:port)Live private IPImageArchCPU / Mem
skyhub-prod-api-servicegateway.skyhub-prod.internal:5000<vpc-host-ip>skyhub-prod-api-image:latest (ECR)ARM64256 / 512
skyhub-prod-janus-servicejanus.skyhub-prod.internal:8088/8188/8081172.31.10.110skyhub-prod-janus-image:latest (ECR)X86_64256 / 512
skyhub-prod-whip-servicewhip.skyhub-prod.internal:7080172.31.4.1skyhub-prod-whip-image:latest (ECR)ARM64256 / 512
skyhub-prod-ws-proxy-servicews_proxy.skyhub-prod.internal:7070172.31.2.246skyhub-prod-ws-proxy-image:latest (ECR)ARM64512 / 1024
skyhub-prod-redis-serviceredis.skyhub-prod.internal:6379172.31.0.207public.ecr.aws/docker/library/redis:alpine3.21ARM64256 / 512
skyhub-prod-db-servicedatabase.skyhub-prod.internal:5432172.31.12.178public.ecr.aws/docker/library/postgres:16.3-bullseyeARM64256 / 512
Janus is the only x86 task. modules/janus/janus.tf sets cpu_architecture = "X86_64"; every other service is ARM64 (cheaper Graviton). If you rebuild the Janus image, it must be an amd64 image or the task will fail to start with an exec-format error. ws_proxy is also the odd one out on resources — 512 CPU / 1024 MB, double the others (modules/our_ws_proxy/proxy.tf:137).

Shared task-definition conventions

Every module follows the same shape, so once you know one you know all six:
  • network_mode = "awsvpc", requires_compatibilities = ["FARGATE"], desired_count = 1, launch_type = "FARGATE".
  • The single container is always named container-definition — you need this name for aws ecs execute-command and for the service_registries block.
  • enable_execute_command = true on all six, so you can shell into any task (see Operating the services).
  • Logs go to CloudWatch group /ecs/<tag>-<service>-task-definition (the DB group is the exception: /ecs/skyhub-prod-db-task). api, whip, ws_proxy, and redis set retention_in_days = 7; janus and db set no retention (logs never expire).
  • Task and execution roles are the same role per service and are over-privilegedapi and db include a literal Action "*" on Resource "*" statement (modules/api/api.tf:122, modules/database/postgre.tf:95). A least-privilege pass is outstanding.
  • Only janus, whip, ws_proxy, and redis attach a Cloud Map health_check_custom_config (failure threshold 1); api and database have it commented out.

Per-service detail

modules/api/api.tf — port 5000/tcp, registered as gateway.skyhub-prod.internal. This is the only service with a large hand-written env block plus three SSM secrets (SSH_PRIVATE_KEY, SSH_PUBLIC_KEY, SSH_KNOWN_HOSTS from arn:aws:ssm:.../skyhub-prod/ssh/*) that the entrypoint uses to reach the on-prem Docker host. Infra-relevant env wiring baked into the task definition:
Env varValue in api.tfPurpose
DB_IPdatabase.skyhub-prod.internalPostgres host (Cloud Map)
JANUS_URLhttp://janus.skyhub-prod.internal:8088/janusVideo room creation
WHIP_SERVER_URLhttp://whip.skyhub-prod.internal:7080WebRTC ingest
VPN_SERVICE_IP / VPN_SERVICE_PORTjumphost-private... / 5050Per-user VPN service
JUMPHOST_IP / JUMPHOST_PORTjumphost-private... / 9090Rosbridge proxy to drones
DOCKER_HOST / REMOTE_DOCKER_HOSTtcp://jumphost-private.skyhub-prod.internal:2375SITL container mgmt
DOCKER_HOST_IP<office-docker-host>Direct rosbridge over WireGuard
DEPLOYMENT_ENVIRONMENTserverDisables insecure dev defaults
ENABLE_SITLtrueEnables SITL orchestration
SITL_VIDEO_STREAM_DRONE_STATECONNECTEDWhen SITL video starts
JWT_SECRET_KEY and the Gmail MAIL_PASSWORD are committed as plaintext env literals in modules/api/api.tf:226 and :237. They should be moved to SSM/Secrets Manager and rotated. The live task definition also carries additional env (Stripe, OTEL, telemetry throttle) not present in api.tf — the ECS task def and the Terraform have drifted. Treat Gateway Environment Variables and AWS Production Configuration as the reconciled reference.
The two Docker paths are deliberate and must be preserved: :2375 (via jumphost nginx) for the Docker Engine API, and <office-docker-host> directly over WireGuard for rosbridge telemetry. See SITL Drone Lifecycle and Rosbridge Connection.
modules/janus/janus.tf — the only X86_64 task. TCP ports 8088 (HTTP/janus API), 8188 (WebSocket signaling), 8081 (admin UI). The security group additionally opens UDP 10000-61000 for WebRTC media, and the task advertises the jumphost public EIP via GATEWAY_IP. Key env: STUN_SERVER=stun.l.google.com, STUN_PORT=19302, RTP_PORT_RANGE=20000-20099, WEBSOCKETS_ENABLED=true. Deep dive: Janus WebRTC SFU.
modules/whip/whip.tf — ARM64, port 7080/tcp. Bridges drone H264 ingest into Janus rooms. Env: JANUS_ADDRESS=ws://janus.skyhub-prod.internal:8188, GATEWAY_SERVICE_IP=gateway.skyhub-prod.internal, GATEWAY_SERVICE_PORT=5000. Deep dive: WHIP Ingest Server.
modules/our_ws_proxy/proxy.tf — ARM64, 512 CPU / 1024 MB (the largest task), port 7070/tcp. Bridges the low-latency manual-control path to Redis pub/sub and reads Postgres to resolve drone IPs. Env: REDIS_HOST=redis.skyhub-prod.internal, DB_HOST=database.skyhub-prod.internal, plus DB_NAME/DB_USER/DB_PASSWORD. Deep dive: WebSocket Gamepad Proxy.

Postgres runs on Fargate + EFS (not RDS)

This is the single most surprising fact about the deployment: database.skyhub-prod.internal is a postgres:16.3-bullseye container, not an RDS instance (modules/database/postgre.tf).
1

One container, one task

desired_count = 1, ARM64, 256 CPU / 512 MB. Port 5432. Credentials are hardcoded in the container env: POSTGRES_USER=idrobots, POSTGRES_PASSWORD=idrobots, POSTGRES_DB=skyhub (postgre.tf:257). The same weak credentials are reused by the gateway task and the migration CodeBuild.
2

Durability via EFS, not RDS snapshots

An aws_efs_file_system (skyhub-prod-postgres-efs) is mounted into the task as the postgres-data volume at /var/lib/postgresql/data via an EFS access point (uid/gid 0, transit_encryption = "ENABLED"). If the Fargate task is replaced, the new task re-mounts the same EFS and the data survives.
3

Backups via AWS Backup

modules/database/backup.tf adds a daily EFS backup (14-day retention, midnight UTC) plus a daily EC2 backup (7-day retention, tag Backup=daily → the jumphost). There are no read replicas and no Multi-AZ — durability rests entirely on EFS + AWS Backup. Full detail on Storage, Backups & Alerting.
Because there is no RDS, a desired_count = 1 DB task on a single AZ is a hard availability floor for the whole platform. Any docs or code referring to an “RDS endpoint” (including the older docs/AWS_PRODUCTION_CONFIG.md) is stale — the real endpoint is the Cloud Map name database.skyhub-prod.internal:5432. Migrations run against it via flask db upgrade; see Migrations, DB Connection & Dev Mode.

Redis is ephemeral (not ElastiCache)

modules/our_redis/redis.tf runs public.ecr.aws/docker/library/redis:alpine3.21, ARM64, port 6379, as redis.skyhub-prod.internal. It brokers gamepad/core pub-sub and can optionally back Socket.IO horizontal scaling.
  • No persistence volume — a task replacement wipes everything. Treat it strictly as a cache / message bus, never as a store of record.
  • No auth password is configured on the container. It is reachable only inside the private subnet + security groups; do not expose it.

API redeploy-loop CloudWatch alarms

Only the API/gateway service has alarms. They exist specifically to catch restart/redeploy loops (a task that keeps crashing and re-launching), and all three publish to the shared SNS topic → Slack notifier Lambda + email (modules/api/api.tf, alarm_topic_arn from modules/alarms).
AlarmMetric (namespace)ConditionMeaning
skyhub-prod-API-Pending-Tasks-HighPendingTaskCount (ECS/ContainerInsights)Avg > 0.3 for 3× 5-min periodsTasks stuck pending for 15+ min → launch loop likely
skyhub-prod-API-Multiple-DeploymentsDeploymentCount (ECS/ContainerInsights)Avg > 1.4 for 4× 2-min periodsMore than one active deployment for 8+ min → rollout not settling
skyhub-prod-API-Shutting-Down-Log-SpikeAPI-ShuttingDownCount (skyhub-prod/ECSLogs)Sum > 4 for 3× 4-min periodsMore than 4 "Shutting down" log lines in 12 min → crash loop
The third alarm is fed by a log metric filter on the API log group that counts occurrences of the literal string Shutting down (api.tf:406). If you see it fire, tail the logs first — the gateway prints Shutting down on graceful worker exits, so a spike usually means gunicorn is being killed and restarted repeatedly (frequently an OOM at 512 MB, a failed config validation, or a bad :latest image push).
These alarms are the reason the gateway does not run on FARGATE_SPOT — the launch_type = "FARGATE" pin keeps the single API task off spot reclamation, so an alarm firing genuinely indicates a bad deploy rather than routine interruption.

Operating the services

Real commands against the live cluster (region eu-central-1):
# List all services and their running/desired counts
aws ecs list-services --cluster skyhub-prod-cluster --region eu-central-1
aws ecs describe-services --cluster skyhub-prod-cluster \
  --services skyhub-prod-api-service --region eu-central-1 \
  --query 'services[0].{running:runningCount,desired:desiredCount,deployments:deployments}'

# Tail a service's logs (retention: 7d for api/whip/ws-proxy/redis; unlimited for janus/db)
aws logs tail /ecs/skyhub-prod-api-task-definition --follow --region eu-central-1

# Shell into a running task (container is always named "container-definition")
aws ecs execute-command --cluster skyhub-prod-cluster \
  --task <task-id> --container container-definition \
  --interactive --command "/bin/sh" --region eu-central-1

# Force a fresh deployment (what CI/CD does after pushing a new :latest image)
aws ecs update-service --cluster skyhub-prod-cluster \
  --service skyhub-prod-api-service --force-new-deployment --region eu-central-1
Images are deployed as the mutable :latest tag; a git tag v_* triggers CodeBuild → ECR push → update-service --force-new-deployment. See CI/CD: CodeBuild, ECR & Frontend Deploy and the gateway-specific packaging on Gateway Build, Docker & Runtime.

Gotchas a refactor must preserve

  • Cloud Map names are load-bearing. gateway/janus/whip/ws_proxy/redis/database.skyhub-prod.internal are baked into the gateway env, the ws_proxy env, the WHIP env, and the jumphost nginx config. Renaming a service or the namespace breaks all of them at once.
  • The container name is always container-definition. It’s referenced by service_registries and by every execute-command. Renaming it silently breaks service discovery registration.
  • Janus must stay x86; the other five must stay ARM64 to match their built images.
  • launch_type = "FARGATE" per service overrides the cluster’s 100% Spot default — this is intentional, not a bug.
  • Single AZ, single NAT, desired_count = 1 everywhere. None of these services is highly available; Postgres-on-EFS and the jumphost are the hard SPOFs.
  • Postgres/Redis are containers. Anyone reaching for “the RDS instance” or “ElastiCache” is looking at the wrong mental model.