skyhub-prod-cluster, in eu-central-1 (account <aws-account-id>). There is no RDS instance, no ElastiCache, and no load balancer in the account — Postgres and Redis are ordinary containers on the same cluster, and all public ingress arrives through the WireGuard/nginx jumphost rather than an ALB (see VPC, WireGuard Jumphost & nginx Routing).
Each service is a terraform-aws-module-style module under skyhub_terraform/modules/ that creates one aws_ecs_task_definition + one aws_ecs_service (desired count 1) and registers an A record in AWS Cloud Map under the private DNS namespace skyhub-prod.internal (modules/ecs/ecs.tf). Services find each other exclusively by these *.skyhub-prod.internal names — there are no hardcoded task IPs.
This page documents the task definitions: resources, architecture, ports, env wiring, and the API redeploy-loop alarms. For the cluster/VPC big picture see AWS Infrastructure Overview; for the nginx routing that fronts these ports see Networking & Jumphost; for the full gateway env-var catalog see Gateway Environment Variables.
The cluster
skyhub_terraform/modules/ecs/ecs.tf
- Container Insights is on, which is what makes the
ECS/ContainerInsightsmetrics (used by the API alarms below) available. - The cluster’s default strategy is 100%
FARGATE_SPOT, but every service pinslaunch_type = "FARGATE", which overrides that default and places tasks on standard on-demand Fargate. To actually move a service onto Spot you must droplaunch_typeand add acapacity_provider_strategyblock — changing only the cluster default has no effect. - There are 0 registered container instances — this is pure serverless Fargate.
Service inventory
The private IPs below are the live values captured from prod (skyhub-prod.internal Cloud Map zone); they change on every task replacement, so always resolve the DNS name, never the IP.
| ECS service | Cloud Map DNS (:port) | Live private IP | Image | Arch | CPU / Mem |
|---|---|---|---|---|---|
skyhub-prod-api-service | gateway.skyhub-prod.internal:5000 | <vpc-host-ip> | skyhub-prod-api-image:latest (ECR) | ARM64 | 256 / 512 |
skyhub-prod-janus-service | janus.skyhub-prod.internal:8088/8188/8081 | 172.31.10.110 | skyhub-prod-janus-image:latest (ECR) | X86_64 | 256 / 512 |
skyhub-prod-whip-service | whip.skyhub-prod.internal:7080 | 172.31.4.1 | skyhub-prod-whip-image:latest (ECR) | ARM64 | 256 / 512 |
skyhub-prod-ws-proxy-service | ws_proxy.skyhub-prod.internal:7070 | 172.31.2.246 | skyhub-prod-ws-proxy-image:latest (ECR) | ARM64 | 512 / 1024 |
skyhub-prod-redis-service | redis.skyhub-prod.internal:6379 | 172.31.0.207 | public.ecr.aws/docker/library/redis:alpine3.21 | ARM64 | 256 / 512 |
skyhub-prod-db-service | database.skyhub-prod.internal:5432 | 172.31.12.178 | public.ecr.aws/docker/library/postgres:16.3-bullseye | ARM64 | 256 / 512 |
Shared task-definition conventions
Every module follows the same shape, so once you know one you know all six:network_mode = "awsvpc",requires_compatibilities = ["FARGATE"],desired_count = 1,launch_type = "FARGATE".- The single container is always named
container-definition— you need this name foraws ecs execute-commandand for theservice_registriesblock. enable_execute_command = trueon all six, so you can shell into any task (see Operating the services).- Logs go to CloudWatch group
/ecs/<tag>-<service>-task-definition(the DB group is the exception:/ecs/skyhub-prod-db-task).api,whip,ws_proxy, andredissetretention_in_days = 7;janusanddbset no retention (logs never expire). - Task and execution roles are the same role per service and are over-privileged —
apianddbinclude a literalAction "*"onResource "*"statement (modules/api/api.tf:122,modules/database/postgre.tf:95). A least-privilege pass is outstanding. - Only
janus,whip,ws_proxy, andredisattach a Cloud Maphealth_check_custom_config(failure threshold 1);apianddatabasehave it commented out.
Per-service detail
api / gateway — the Flask/Socket.IO control plane
api / gateway — the Flask/Socket.IO control plane
modules/api/api.tf — port 5000/tcp, registered as gateway.skyhub-prod.internal. This is the only service with a large hand-written env block plus three SSM secrets (SSH_PRIVATE_KEY, SSH_PUBLIC_KEY, SSH_KNOWN_HOSTS from arn:aws:ssm:.../skyhub-prod/ssh/*) that the entrypoint uses to reach the on-prem Docker host. Infra-relevant env wiring baked into the task definition:| Env var | Value in api.tf | Purpose |
|---|---|---|
DB_IP | database.skyhub-prod.internal | Postgres host (Cloud Map) |
JANUS_URL | http://janus.skyhub-prod.internal:8088/janus | Video room creation |
WHIP_SERVER_URL | http://whip.skyhub-prod.internal:7080 | WebRTC ingest |
VPN_SERVICE_IP / VPN_SERVICE_PORT | jumphost-private... / 5050 | Per-user VPN service |
JUMPHOST_IP / JUMPHOST_PORT | jumphost-private... / 9090 | Rosbridge proxy to drones |
DOCKER_HOST / REMOTE_DOCKER_HOST | tcp://jumphost-private.skyhub-prod.internal:2375 | SITL container mgmt |
DOCKER_HOST_IP | <office-docker-host> | Direct rosbridge over WireGuard |
DEPLOYMENT_ENVIRONMENT | server | Disables insecure dev defaults |
ENABLE_SITL | true | Enables SITL orchestration |
SITL_VIDEO_STREAM_DRONE_STATE | CONNECTED | When SITL video starts |
:2375 (via jumphost nginx) for the Docker Engine API, and <office-docker-host> directly over WireGuard for rosbridge telemetry. See SITL Drone Lifecycle and Rosbridge Connection.janus — WebRTC SFU (x86)
janus — WebRTC SFU (x86)
modules/janus/janus.tf — the only X86_64 task. TCP ports 8088 (HTTP/janus API), 8188 (WebSocket signaling), 8081 (admin UI). The security group additionally opens UDP 10000-61000 for WebRTC media, and the task advertises the jumphost public EIP via GATEWAY_IP. Key env: STUN_SERVER=stun.l.google.com, STUN_PORT=19302, RTP_PORT_RANGE=20000-20099, WEBSOCKETS_ENABLED=true. Deep dive: Janus WebRTC SFU.whip — WebRTC-HTTP ingest
whip — WebRTC-HTTP ingest
modules/whip/whip.tf — ARM64, port 7080/tcp. Bridges drone H264 ingest into Janus rooms. Env: JANUS_ADDRESS=ws://janus.skyhub-prod.internal:8188, GATEWAY_SERVICE_IP=gateway.skyhub-prod.internal, GATEWAY_SERVICE_PORT=5000. Deep dive: WHIP Ingest Server.ws_proxy — gamepad WebSocket proxy
ws_proxy — gamepad WebSocket proxy
modules/our_ws_proxy/proxy.tf — ARM64, 512 CPU / 1024 MB (the largest task), port 7070/tcp. Bridges the low-latency manual-control path to Redis pub/sub and reads Postgres to resolve drone IPs. Env: REDIS_HOST=redis.skyhub-prod.internal, DB_HOST=database.skyhub-prod.internal, plus DB_NAME/DB_USER/DB_PASSWORD. Deep dive: WebSocket Gamepad Proxy.Postgres runs on Fargate + EFS (not RDS)
This is the single most surprising fact about the deployment:database.skyhub-prod.internal is a postgres:16.3-bullseye container, not an RDS instance (modules/database/postgre.tf).
One container, one task
desired_count = 1, ARM64, 256 CPU / 512 MB. Port 5432. Credentials are hardcoded in the container env: POSTGRES_USER=idrobots, POSTGRES_PASSWORD=idrobots, POSTGRES_DB=skyhub (postgre.tf:257). The same weak credentials are reused by the gateway task and the migration CodeBuild.Durability via EFS, not RDS snapshots
An
aws_efs_file_system (skyhub-prod-postgres-efs) is mounted into the task as the postgres-data volume at /var/lib/postgresql/data via an EFS access point (uid/gid 0, transit_encryption = "ENABLED"). If the Fargate task is replaced, the new task re-mounts the same EFS and the data survives.Backups via AWS Backup
modules/database/backup.tf adds a daily EFS backup (14-day retention, midnight UTC) plus a daily EC2 backup (7-day retention, tag Backup=daily → the jumphost). There are no read replicas and no Multi-AZ — durability rests entirely on EFS + AWS Backup. Full detail on Storage, Backups & Alerting.Redis is ephemeral (not ElastiCache)
modules/our_redis/redis.tf runs public.ecr.aws/docker/library/redis:alpine3.21, ARM64, port 6379, as redis.skyhub-prod.internal. It brokers gamepad/core pub-sub and can optionally back Socket.IO horizontal scaling.
- No persistence volume — a task replacement wipes everything. Treat it strictly as a cache / message bus, never as a store of record.
- No auth password is configured on the container. It is reachable only inside the private subnet + security groups; do not expose it.
API redeploy-loop CloudWatch alarms
Only the API/gateway service has alarms. They exist specifically to catch restart/redeploy loops (a task that keeps crashing and re-launching), and all three publish to the shared SNS topic → Slack notifier Lambda + email (modules/api/api.tf, alarm_topic_arn from modules/alarms).
| Alarm | Metric (namespace) | Condition | Meaning |
|---|---|---|---|
skyhub-prod-API-Pending-Tasks-High | PendingTaskCount (ECS/ContainerInsights) | Avg > 0.3 for 3× 5-min periods | Tasks stuck pending for 15+ min → launch loop likely |
skyhub-prod-API-Multiple-Deployments | DeploymentCount (ECS/ContainerInsights) | Avg > 1.4 for 4× 2-min periods | More than one active deployment for 8+ min → rollout not settling |
skyhub-prod-API-Shutting-Down-Log-Spike | API-ShuttingDownCount (skyhub-prod/ECSLogs) | Sum > 4 for 3× 4-min periods | More than 4 "Shutting down" log lines in 12 min → crash loop |
Shutting down (api.tf:406). If you see it fire, tail the logs first — the gateway prints Shutting down on graceful worker exits, so a spike usually means gunicorn is being killed and restarted repeatedly (frequently an OOM at 512 MB, a failed config validation, or a bad :latest image push).
Operating the services
Real commands against the live cluster (regioneu-central-1):
Images are deployed as the mutable
:latest tag; a git tag v_* triggers CodeBuild → ECR push → update-service --force-new-deployment. See CI/CD: CodeBuild, ECR & Frontend Deploy and the gateway-specific packaging on Gateway Build, Docker & Runtime.Gotchas a refactor must preserve
- Cloud Map names are load-bearing.
gateway/janus/whip/ws_proxy/redis/database.skyhub-prod.internalare baked into the gateway env, the ws_proxy env, the WHIP env, and the jumphost nginx config. Renaming a service or the namespace breaks all of them at once. - The container name is always
container-definition. It’s referenced byservice_registriesand by everyexecute-command. Renaming it silently breaks service discovery registration. - Janus must stay x86; the other five must stay ARM64 to match their built images.
launch_type = "FARGATE"per service overrides the cluster’s 100% Spot default — this is intentional, not a bug.- Single AZ, single NAT,
desired_count = 1everywhere. None of these services is highly available; Postgres-on-EFS and the jumphost are the hard SPOFs. - Postgres/Redis are containers. Anyone reaching for “the RDS instance” or “ElastiCache” is looking at the wrong mental model.

