AWS Production Configuration

This page is the runtime-configuration reference for the Gateway Service in production — the environment variables the ECS Fargate task actually runs with, how the gateway brokers image-pull credentials to physical drones, and how it drives SITL containers on the remote office Docker host. It does not re-document the AWS topology (see /deployment/infra-overview), the jumphost/nginx ingress (see /deployment/networking-jumphost), or the CI/CD image pipeline (see /deployment/cicd-images).

Production runs in AWS eu-central-1, account <aws-account-id>, as a single ECS Fargate service skyhub-prod-api-service registered in Cloud Map as gateway.skyhub-prod.internal:5000. The service framing is DEPLOYMENT_ENVIRONMENT=server. Postgres and Redis are containers on Fargate (not RDS/ElastiCache), and the SITL/gamepad/core drone containers run on the on-prem office server nexus0 (<office-docker-host>) reached over WireGuard.

Source of truth for prod config

The live configuration comes from three places that must be kept reconciled:

Source	Path	What it is
Live ECS task env	`docs/aws_prod_env_vars.env`	Vars exported from `skyhub-prod-api-task-definition:46` (2026-01-28) — what is actually deployed
Config playbook	`docs/AWS_PRODUCTION_CONFIG.md`	Human-written required-vars + checklist
Terraform task def	`skyhub_terraform` `modules/api/api.tf`	The `environment[]` block + SSM `secrets[]` that generate the task definition

Every var is read in src/application/settings.py; the full catalogue with per-var descriptions lives in /gateway/config/environment-variables.

docs/aws_prod_env_vars.env in the repo contains real live secret values — the JWT signing key, the Gmail app password, and live Stripe keys (sk_live_…, pk_live_…, whsec_…). modules/api/api.tf additionally hardcodes JWT_SECRET_KEY and MAIL_PASSWORD in plaintext. Treat all of these as compromised: rotate them and move them to SSM Parameter Store / Secrets Manager. Every value below is redacted; never paste the committed values into docs, tickets, or logs.

Production environment variables

The block below is the deployed task-definition env with all secrets replaced by placeholders. Non-secret infra facts (DNS names, buckets, IPs, ports) are shown verbatim.

aws_prod_env_vars.env (skyhub-prod-api-task-definition:46, redacted)

# --- Deployment identity ---
DEPLOYMENT_ENVIRONMENT=server          # must be 'server'; gates insecure dev defaults
ENVIRONMENT_STAGE=prod
RESOURCE_TAG=skyhub-prod               # drives SSM paths, image URLs, resource names
ACCOUNT_ID=<aws-account-id>
REGION=eu-central-1

# --- Database (Postgres container on EFS, via Cloud Map) ---
DB_NAME=skyhub
DB_USERNAME=idrobots
DB_PASSWORD=<redacted>                 # weak/shared; matches container POSTGRES_PASSWORD
DB_IP=database.skyhub-prod.internal

# --- Auth ---
JWT_SECRET_KEY=<redacted>              # REQUIRED; startup hard-fails without it (non-local)

# --- SITL / remote Docker (see section below) ---
ENABLE_SITL=true
SITL_IMAGE_NAME=<aws-account-id>.dkr.ecr.eu-central-1.amazonaws.com/skyhub-sitl-on-prem-dev
SITL_CPU_LIMIT=1.0
SITL_MEMORY_LIMIT=4g
SITL_VIDEO_STREAM_DRONE_STATE=CONNECTED
USER_SITL_MAX_COUNT=5
REMOTE_DOCKER_ENABLED=true
REMOTE_DOCKER_HOST=tcp://jumphost-private.skyhub-prod.internal:2375
REMOTE_DOCKER_SSH_TARGET=ssh://nexus0@<office-docker-host>
DOCKER_HOST_IP=<office-docker-host>

# --- Video ---
JANUS_URL=http://janus.skyhub-prod.internal:8088/janus
WHIP_SERVER_URL=http://whip.skyhub-prod.internal:7080

# --- VPN / jumphost routing ---
VPN_SERVICE_IP=jumphost-private.skyhub-prod.internal
VPN_SERVICE_PORT=5050
VPN_BUCKET=skyhub-prod-user-vpn
JUMPHOST_IP=jumphost-private.skyhub-prod.internal
JUMPHOST_PORT=9090

# --- Email (Gmail SMTP) ---
MAIL_SERVER=smtp.gmail.com
MAIL_PORT=587
MAIL_USERNAME=[email protected]
MAIL_PASSWORD=<redacted>

# --- Assets ---
ASSET_BUCKET=skyhub-prod-assets

# --- Stripe (LIVE) ---
STRIPE_SECRET_KEY=sk_live_xxx
STRIPE_PUBLISHABLE_KEY=pk_live_xxx
STRIPE_WEBHOOK_SECRET=whsec_xxx
STRIPE_PRICE_ID=price_xxx
STRIPE_SUCCESS_URL=https://skyhub.ai/billing/success
STRIPE_CANCEL_URL=https://skyhub.ai/billing/cancel

# --- Observability (OTel -> SigNoz) ---
OTEL_EXPORTER_OTLP_ENDPOINT=http://jumphost-private.skyhub-prod.internal:4317
OTEL_SERVICE_NAME=skyhub_gateway_service
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod

# --- App ---
FLASK_APP=main.py
SOCKET_IP=0.0.0.0
LOG_LEVEL=10                           # 10 = DEBUG
SEND_QUEUE_SIZE=200
ENABLE_REGISTRATION=true

Config drift to watch

The live rev-46 task definition is missing several vars the code and playbook expect. When they are absent the gateway silently falls back to the settings.py defaults shown, which are wrong for prod:

Missing var	settings.py fallback	Should be (prod)
`GAMEPAD_IMAGE_NAME`	`skyhub-gamepad:latest`	`…ecr…/skyhub-gamepad:latest`
`GAMEPAD_CPU_LIMIT` / `GAMEPAD_MEMORY_LIMIT`	`0.5` / `512m`	`2` / `2g`
`GAMEPAD_API_URL`	`http://localhost:5000`	`https://prod.skyhub.ai:5000`
`CORE_CPU_LIMIT` / `CORE_MEMORY_LIMIT`	`2.0` / `2g`	`4` / `4g`
`SITL_REDIS_HOST`	`host.docker.internal`	`redis.skyhub-prod.internal`
`REDIS_PASSWORD`	`skyhub_redis_secret`	`<redacted>`
`LOG_ERASE_AFTER_DOWNLOAD`	`false`	`true`
`OTEL_EXPORTER_OTLP_HEADERS`	unset	`signoz-access-token=<redacted>`

GAMEPAD_API_URL is the URL the on-drone gamepad containers call back into for execution tracking. It must be reachable from the office server <office-docker-host>. Because the fallback is http://localhost:5000, an unset value does not fail drone creation — it silently breaks execution/log callbacks. Also note SITL_CPU_LIMIT=1.0 in the live task def vs 4 in AWS_PRODUCTION_CONFIG.md, and SITL_IMAGE_NAME points at skyhub-sitl-on-prem-dev rather than the skyhub-sitl:local the checklist mentions.

Secrets and SSM injection

Non-secret env vars are inlined in the task definition. Three SSH secrets are injected as ECS secrets[] from SSM Parameter Store (modules/api/api.tf:268), which the container entrypoint materializes into /root/.ssh:

Task env var	SSM parameter	Purpose
`SSH_PRIVATE_KEY`	`/skyhub-prod/ssh/private-key`	Key to SSH the office Docker host
`SSH_PUBLIC_KEY`	`/skyhub-prod/ssh/public-key`	Matching public key
`SSH_KNOWN_HOSTS`	`/skyhub-prod/ssh/known-hosts`	Pinned host keys

containers/docker-entrypoint.sh writes these files, then configures an SSH ProxyCommand through the jumphost (JUMPHOST_PUBLIC_IP, default <prod-ingress-ip>) to reach nexus0@<office-docker-host>, and — when REMOTE_DOCKER_ENABLED=true and REMOTE_DOCKER_SSH_TARGET is set — opens an ssh -f -N -L 2375:localhost:2375 tunnel to the remote Docker daemon before execing gunicorn.

REMOTE_DOCKER_SSH_TARGET is read only by the shell entrypoint, never by settings.py. The Python SITLDroneService connects to REMOTE_DOCKER_HOST (the tcp://…:2375 endpoint that the tunnel/jumphost exposes). Both must agree or SITL container control breaks. See /deployment/gateway-build-runtime for the entrypoint and gunicorn details.

For everything else (S3 assets, presigned URLs, ECR, SSM, STS) the ECS task uses its task role — there are no static AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY in the prod task definition (those exist only in .env for local dev).

The ECR registry

The private registry is <aws-account-id>.dkr.ecr.eu-central-1.amazonaws.com, read from ECR_REPO and surfaced to callers as settings.DOCKER_REPO (src/application/settings.py:85). It hosts:

Application images — skyhub-prod-api-image (this gateway), skyhub-prod-janus-image, skyhub-prod-whip-image, skyhub-prod-ws-proxy-image, skyhub-prod-vpn-image.
Drone (core) layered images — skyhub-prod-drone-base-image → -basebuild- → -ros2- → -ws- → -mavp2p- → -rtsp-image, plus skyhub-core, skyhub-gamepad.
SITL images — skyhub-sitl-on-prem-dev, sitl-ardupilot-on-prem-dev, and related on-prem variants.

The gateway authenticates to ECR in two ways: boto3 ecr.get_authorization_token() (for the drone activation flow) and a shell-out to aws ecr get-login-password --region eu-central-1 (for remote-Docker image pulls, src/service/sitl_drone_service.py:50).

Credential brokering to physical drones

A physical drone owns no long-lived AWS credentials. The gateway brokers short-lived ones over two endpoints in src/routes/drone_routes.py.

`GET /drone/activate`

Authenticated by a 10-digit token header. The handler (src/routes/drone_routes.py:1527) waits up to ~15s for the drone’s WireGuard peer to come up, then returns a bundle so the drone can bootstrap itself:

Field	Source	Notes
`username` / `password`	`ecr.get_authorization_token()` (base64-decoded)	Short-lived ECR login
`repository`	`settings.DOCKER_REPO`	The ECR registry host
`vpn`	User VPN service	Per-user WireGuard config link
`compose`	S3 presigned URL (`ExpiresIn=300`)	See INSTALLER_BUCKET below

The activation token is cleared after a successful call (physical_drone_service.activate).

`GET /drone/pull`

Guarded by the @check_vpn_ip middleware — the caller is trusted purely by its 10.71.x WireGuard source IP; the drone is resolved via get_drone_by_ip(request.vpn_ip). The handler (src/routes/drone_routes.py:1638) reads the role ARN from SSM /{RESOURCE_TAG}/pull_role (i.e. /skyhub-prod/pull_role, a read-only-ECR IAM role, modules/drone/roles.tf:52) and sts.assume_role(...) for a 1-hour session, returning temporary accessKeyId/secretAccessKey/sessionToken so the drone can pull its own core images from ECR. See /gateway/security/vpn-middleware-jumphost for the VPN-IP auth model.

INSTALLER_BUCKET compose distribution

INSTALLER_BUCKET (default skyhubcore, src/application/settings.py:86) holds the drone-side docker-compose files that the Terraform drone pipeline templates and uploads. On activation the gateway picks the file by environment and hands the drone a presigned GET URL valid for 300s:

src/routes/drone_routes.py:1605

if settings.DEPLOYMENT_ENVIRONMENT.startswith("dev"):
    compose_file = "docker-compose.yml"
else:
    compose_file = "docker-compose.prod.yml"

In prod (DEPLOYMENT_ENVIRONMENT=server) the drone therefore receives s3://skyhubcore/docker-compose.prod.yml. The drone downloads it via the presigned URL, logs into ECR with the brokered credentials, and pulls its layered core images.

SITL remote-Docker flow

SITL drones do not run in AWS. With ENABLE_SITL=true and REMOTE_DOCKER_ENABLED=true, SITLDroneService (src/service/sitl_drone_service.py) manages containers on the office server through two distinct paths that any refactor must preserve:

Container management

Docker Engine API at REMOTE_DOCKER_HOST=tcp://jumphost-private.skyhub-prod.internal:2375. nginx on the jumphost forwards :2375 to <office-docker-host>:2375 over WireGuard. Images are pulled from ECR using the token from aws ecr get-login-password.

Telemetry / rosbridge

The gateway opens rosbridge WebSockets directly to DOCKER_HOST_IP=<office-docker-host> over the WireGuard tunnel — not through the :2375 proxy — treating the SITL container like a physical drone.

The Docker client is chosen at init (src/service/sitl_drone_service.py:37): docker.DockerClient(base_url=REMOTE_DOCKER_HOST, use_ssh_client=…) for remote, or docker.from_env() for local. Container lifecycle, port math, and network-namespace sharing are covered in depth in /gateway/services/sitl-lifecycle.

The prod checklist expects skyhub-sitl:local, skyhub-gamepad:latest, and core:latest to be pre-built and present on the office Docker host — there is no CI job that pushes them there. The gateway only auto-pulls SITL/gamepad/core images from ECR in remote-Docker mode when it has an ecr_auth_config; missing images in local mode raise DroneCreationFailed.

Observability

When OTEL_EXPORTER_OTLP_ENDPOINT is set, initialize_opentelemetry (src/main.py:80) wires OTLP span + log exporters and instruments Flask, Requests, and SQLAlchemy. In prod the endpoint is http://jumphost-private.skyhub-prod.internal:4317; nginx forwards :4317 to the SigNoz collector on the office server at <office-docker-host>:4318. SigNoz auth is a header parsed from OTEL_EXPORTER_OTLP_HEADERS (comma-separated key=value, e.g. signoz-access-token=<redacted>) — that header var is absent from rev-46, so add it when enabling authenticated export. Deep detail: /gateway/observability/opentelemetry-signoz.

Startup validation

The gateway fails fast on missing critical config so a misconfigured task never serves traffic:

settings.py raises RuntimeError at import if DEPLOYMENT_ENVIRONMENT != local and JWT_SECRET_KEY is empty (src/application/settings.py:118).
validate_critical_config() (src/main.py:137) raises if any of REGION, DB_IP, JWT_SECRET_KEY are missing — and additionally requires VPN_BUCKET when not local.

See /gateway/config/startup-validation for the full boot sequence.

Pre-deployment checklist

Set the deployment identity

DEPLOYMENT_ENVIRONMENT=server, RESOURCE_TAG=skyhub-prod, ACCOUNT_ID=<aws-account-id>, REGION=eu-central-1.

Provision secrets (do NOT inline)

Set a strong JWT_SECRET_KEY; move it, MAIL_PASSWORD, DB creds, Stripe keys, and the SigNoz token to SSM/Secrets Manager. Rotate any value that was ever committed to the repo.

Wire the database and buckets

DB_IP=database.skyhub-prod.internal, DB_NAME=skyhub, credentials; ASSET_BUCKET=skyhub-prod-assets, VPN_BUCKET=skyhub-prod-user-vpn, INSTALLER_BUCKET=skyhubcore. The task role (not static keys) grants S3/ECR/SSM/STS.

Inject the SSH secrets from SSM

/skyhub-prod/ssh/{private-key,public-key,known-hosts} → SSH_PRIVATE_KEY / SSH_PUBLIC_KEY / SSH_KNOWN_HOSTS. Confirm the entrypoint can reach nexus0@<office-docker-host> via the jumphost.

Configure remote Docker / SITL

REMOTE_DOCKER_ENABLED=true, REMOTE_DOCKER_HOST=tcp://jumphost-private.skyhub-prod.internal:2375, REMOTE_DOCKER_SSH_TARGET=ssh://nexus0@<office-docker-host>, DOCKER_HOST_IP=<office-docker-host>. Pre-build/push skyhub-sitl, skyhub-gamepad, core images to the office host.

Fill the missing drift vars

Add GAMEPAD_API_URL=https://prod.skyhub.ai:5000, GAMEPAD_*/CORE_* limits and images, SITL_REDIS_HOST=redis.skyhub-prod.internal, REDIS_PASSWORD, LOG_ERASE_AFTER_DOWNLOAD=true (see the drift table above).

Configure Stripe (if billing enabled)

STRIPE_SECRET_KEY, STRIPE_PUBLISHABLE_KEY, STRIPE_WEBHOOK_SECRET, STRIPE_PRICE_ID, success/cancel URLs. Detail in /gateway/services/billing.

Set observability + video

OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, OTEL_SERVICE_NAME; JANUS_URL, WHIP_SERVER_URL.

Verify VPN connectivity

Confirm the User VPN service (VPN_SERVICE_IP:VPN_SERVICE_PORT = jumphost-private.skyhub-prod.internal:5050) responds so /drone/activate can hand out WireGuard configs. See /ecosystem/user-vpn.

Deploys are triggered by pushing a git tag matching ^refs/tags/v_.*$, which fires CodeBuild → ECR :latest → ecs update-service --force-new-deployment. That pipeline, plus DB-migration builds, is documented in /deployment/cicd-images. The image build itself (Dockerfile.ecr, gunicorn runtime) is in /deployment/gateway-build-runtime.

​Source of truth for prod config

​Production environment variables

​Config drift to watch

​Secrets and SSM injection

​The ECR registry

​Credential brokering to physical drones

​GET /drone/activate

​GET /drone/pull

​INSTALLER_BUCKET compose distribution

​SITL remote-Docker flow

Container management

Telemetry / rosbridge

​Observability

​Startup validation

​Pre-deployment checklist

Source of truth for prod config

Production environment variables

Config drift to watch

Secrets and SSM injection

The ECR registry

Credential brokering to physical drones

`GET /drone/activate`

`GET /drone/pull`

INSTALLER_BUCKET compose distribution

SITL remote-Docker flow

Observability

Startup validation

Pre-deployment checklist