Almost every byte of external traffic into SkyHub production passes through one t4g.nano EC2 instance. It is simultaneously the WireGuard VPN server, the nginx TLS reverse proxy, the drone rosbridge jumphost, and the bridge to the on-prem Docker host. It is the architectural keystone of the whole platform — and its single largest point of failure.
There is no ALB / ELB and no Cloudflare tunnel in production ingress. The skyhub-prod-skyhub-wireguard-server instance (public EIP <prod-ingress-ip>, private <jumphost-private-ip>) is the sole public entrypoint. If it is lost, the UI, REST/Socket.IO API, video signaling, gamepad control, drone rosbridge, remote Docker, and OTLP export all go dark at once. It is backed up only by a daily EC2 AWS Backup plan (tag Backup=daily).

​
The VPC

Defined in environments/prod/vpc_configuration.tf via terraform-aws-modules/vpc/aws v5.15.0. It is deliberately minimal — one AZ, one NAT — with the multi-AZ / database-subnet lines commented out.
PropertyValue
CIDR172.31.0.0/16
AZs1 (azs = [names[0]])
Public subnet<vpc-subnet>/20
Private subnet172.31.0.0/20
NAT gateway1 (single_nat_gateway = true)
AWS VPC resolver172.31.0.2 (used by nginx resolver to look up Cloud Map names)
Service discoveryCloud Map private DNS skyhub-prod.internal
All ECS Fargate services (gateway, janus, whip, ws_proxy, redis, database) run in the private subnet and are reachable only through their Cloud Map A-records (for example gateway.skyhub-prod.internal:5000). The jumphost is the one thing in the public subnet. A single S3 Gateway VPC endpoint (modules/vpc_endpoints) offloads S3 traffic from the lone NAT gateway; ECR interface endpoints are commented out.
This is not an HA topology: single AZ, single NAT, single jumphost, and every ECS service runs desired_count = 1 on FARGATE_SPOT. See ECS Fargate Services and AWS Infrastructure Overview for the service inventory and the reasoning.

​
The WireGuard jumphost instance

Provisioned in modules/wireguard/wireguard.tf.
AttributeValue
Instance typet4g.nano (ARM Graviton)
AMIami-0669c5d030c83a310 (Ubuntu)
Root volume25 GB gp3
Public ENIin public subnet, has Elastic IP <prod-ingress-ip> → Cloud Map jumphost-public.skyhub-prod.internal
Private ENIin private subnet (<jumphost-private-ip>) → Cloud Map jumphost-private.skyhub-prod.internal
SSHTCP 3377 (not 22), key from SSM /skyhub-prod/wireguard_ssh_key
WireGuardUDP 51820 (server 10.69.0.1/16)
The dual-ENI design matters: internal callers use jumphost-private... (the private ENI, which stays inside the VPC and never egresses through the NAT), while browsers/drones use the public EIP. The gateway env deliberately points remote-Docker, VPN, OTLP, and rosbridge routing at jumphost-private.skyhub-prod.internal for exactly this reason.

​
WireGuard address plan

The instance runs the WireGuard server that stitches AWS to the on-prem office network and to physical drones. Three /16 planes are carved out (wireguard.tf variables):
PlaneCIDRPurpose
Infra / server10.69.0.0/16Jumphost (10.69.0.1/.8), office Docker host, SigNoz collector
Users10.70.0.0/16Per-user VPN clients (see User VPN & Network Isolation)
Drones10.71.0.0/16Physical drones; iptables rules per user_drone_access scope reachability
The office/on-prem Docker host is addressed everywhere as <office-docker-host>, but in the static wg_peers list that address is labelled spare-key-1 — the real nexus0 host reuses that peer address. Do not “clean up” that peer without understanding it is load-bearing.

​
The nginx routing table

This is the heart of the page. nginx on the jumphost TLS-terminates the public ports (Let’s Encrypt certs at /cert/fullchain.pem + /cert/privkey.pem, issued by certbot for the frontend domain) and reverse-proxies each listen port to an internal Cloud Map service or an over-VPN host. The authoritative reference is nginx/default; the same config is regenerated at instance boot by environments/prod/wireguard/scripts/post_install_1.tftpl.
Listen (jumphost)TLSUpstreamPurpose
80 / 443yesskyhub-prod-ui-bucket.s3-website.eu-central-1...Static Angular SPA (proxied to the S3 website endpoint)
5000 /socket.io/yesgateway.skyhub-prod.internal:5000Socket.IO telemetry (WebSocket-upgrade headers set)
5000 /yesgateway.skyhub-prod.internal:5000REST API — auth, drones, missions, billing, assets
7070yesws_proxy.skyhub-prod.internal:7070Gamepad WebSocket (WS Proxy)
8188yesjanus.skyhub-prod.internal:8188Janus WebRTC signaling (Janus)
9090nohttp://$http_x_drone_ip:$http_x_drone_portDynamic drone rosbridge passthrough (see below)
2375no<office-docker-host>:2375Remote Docker Engine API on the office host (SITL)
4317no<office-docker-host>:4318OTLP traces → SigNoz collector on the office host
UDP 20000-21100—janus...:$server_portWebRTC media relay (nginx stream {} block)
The Cloud Map upstream locations (5000/7070/8188) each use resolver 172.31.0.2 valid=10s with the host held in a set $upstream_endpoint variable, so nginx re-resolves the Cloud Map A-record on a 10 s TTL rather than pinning an IP at reload — essential because Fargate tasks get new private IPs on every redeploy. Other blocks differ: the 80/443 S3-website block instead uses resolver 8.8.8.8 (public DNS) with the bucket host hardcoded in proxy_pass, 9090 uses neither a resolver nor a set-variable (it proxies to $http_x_drone_ip), and 4317 proxies to a static <office-docker-host> with no resolver.

​
The x-drone-ip / x-drone-port jumphost trick

Port 9090 is a dynamic reverse proxy: nginx forwards to whatever host and port the request declares in its headers, with proxy_pass http://$http_x_drone_ip:$http_x_drone_port; (nginx/default:160). There is no upstream list — the client picks the destination. The gateway is the client. In src/rosbridge/connection.py, when JUMPHOST_IP is set (prod: jumphost-private.skyhub-prod.internal, JUMPHOST_PORT=9090), the Connection object dials the jumphost instead of the drone and attaches the target as headers:
src/rosbridge/connection.py
self.direct = not self.jumphost_ip
...
if not self.direct:
    self.url = f"ws://{self.jumphost_ip}:{self.jumphost_port}"
...
# per-connection websocket handshake (connection.py:162)
header={} if self.direct else {"x-drone-ip": str(self.drone_ip), "x-drone-port": str(self.drone_port)},
If JUMPHOST_IP is empty (local/dev), direct = True and the gateway opens ws://<drone_ip>:9090 straight to the drone with no headers. This one flag makes SITL, physical, and jump-routed drones interchangeable from the connection layer’s perspective.
Port 9090 is effectively an open, unvalidated proxy (an SSRF primitive): nginx will connect to any x-drone-ip/x-drone-port a client supplies. It is safe only because it lives behind the WireGuard network and security group. Never expose 9090 more widely, and never add validation-free header proxying on a new port. The gateway-side counterpart of this trust boundary is documented in VPN IP Authentication & Jumphost Routing.

​
Two separate paths to the office server

The on-prem office host (nexus0 @ <office-docker-host>, reached over WireGuard) is contacted for two unrelated purposes over two different transports. A refactor must preserve this split:

SITL container management

Gateway env REMOTE_DOCKER_HOST / DOCKER_HOST = tcp://jumphost-private.skyhub-prod.internal:2375. nginx :2375 forwards the Docker Engine API to <office-docker-host>:2375. Used to spawn/stop SITL container stacks — see SITL Drone Lifecycle.

Rosbridge telemetry

Gateway env DOCKER_HOST_IP = <office-docker-host>. For SITL drones the gateway opens the rosbridge WebSocket directly to <office-docker-host>:<port> over WireGuard (not via the :9090 header proxy). See Rosbridge Connection & Reconnect.
SSH to nexus0 (REMOTE_DOCKER_SSH_TARGET=ssh://nexus0@<office-docker-host>) uses the three SSH secrets injected from SSM (/skyhub-prod/ssh/...). OTLP traces take a third path: gateway → jumphost-private:4317 → nginx → <office-docker-host>:4318 (SigNoz), covered in OpenTelemetry & SigNoz.

​
Security group exposure

The jumphost SG (modules/wireguard/wireguard.tf) governs what is reachable and from where. Note which ports are open to the world versus VPC-only:
Port(s)ProtoSourceNotes
3377TCP0.0.0.0/0Custom SSH
51820, 51822-51824UDP0.0.0.0/0WireGuard server + client tunnels
80, 443TCP0.0.0.0/0UI / TLS
5000TCP0.0.0.0/0REST + Socket.IO
7070TCP0.0.0.0/0Gamepad WS
8188TCP0.0.0.0/0Janus signaling
9090TCP0.0.0.0/0Rosbridge passthrough
7766TCP0.0.0.0/0Reserved/aux
10000-21000UDP0.0.0.0/0WebRTC media
2375TCPVPC CIDR onlyDocker API (VPC-restricted)
8088TCPVPC CIDR onlyJanus HTTP API
5050TCPVPC CIDR onlyVPN management service (VPN_SERVICE_PORT)
allallegress 0.0.0.0/0Unrestricted egress
The SG opens UDP 10000-21000 but the nginx stream {} block listens on 20000-21100, so 21001-21100 are configured in nginx yet not admitted by the SG. There is also no 4317 ingress rule — OTLP works only because the private ENI sits inside the VPC CIDR reachable by the gateway task’s own SG rules, not because 4317 is explicitly opened here.

​
Config drift & gotchas a future editor must preserve

The checked-in nginx/default (the reference / effectively the running config) contains the 4317 OTLP block but no UDP stream {} media block. The boot template environments/prod/wireguard/scripts/post_install_1.tftpl is the mirror image: it has the UDP media stream but no 4317 block, and it proxies :2375 to <office-host> instead of the prod <office-docker-host>. The live box has clearly been reconciled toward nginx/default; treat that file as source of truth and keep the two in sync when editing routing.
local.frontend_domain is literally "dev.skyhub.ai" in environments/prod/provider_config.tf:69, which is what post_install_1.tftpl feeds to certbot and the server_name. The real prod domain is prod.skyhub.ai (that is what nginx/default and the dashboard’s prod env use). The :2375 server block even carries a stale server_name dev.skyhub.ai. Domain handling is inconsistent between the generated and reference configs.
The routing table is (re)written by a remote-exec provisioner (null_resource.configure_nginx) over SSH on port 3377, not baked into an immutable image. Re-running terraform apply can re-trigger it. Changes made by hand on the box will be clobbered unless mirrored into post_install_1.tftpl / nginx/default.
Upstreams are hardcoded Cloud Map hostnames (gateway/janus/whip/ws_proxy/redis/database/jumphost-private.skyhub-prod.internal). Renaming a service or the namespace simultaneously breaks nginx, the gateway env, and every inter-service call.

​
Adding a new public port

1

Open the port in the SG

Add an aws_security_group_rule in modules/wireguard/wireguard.tf (source 0.0.0.0/0 for public, or var.vpc_cidr_block for VPC-only, matching the existing pattern).
2

Add the nginx server block

Add a matching server { listen <port> ...; } to both nginx/default and environments/prod/wireguard/scripts/post_install_1.tftpl. Use set $upstream_endpoint "<service>.skyhub-prod.internal"; + resolver 172.31.0.2 valid=10s;. Add TLS (ssl_certificate /cert/...) if it is a browser-facing port, and the Upgrade/Connection "upgrade" headers if it carries WebSockets.
3

Re-provision the jumphost

terraform apply re-runs the remote-exec provisioner, or SSH to ubuntu@<prod-ingress-ip> -p 3377 and reload nginx manually (then backport the change so it survives the next apply).
4

Point clients at it

Wire the corresponding gateway/dashboard env var to jumphost-private.skyhub-prod.internal:<port> (internal callers) or prod.skyhub.ai:<port> (browsers). See Gateway Environment Variables and AWS Production Configuration.

Network & VPN Topology

The platform-wide WireGuard mesh and trust boundaries.

AWS Infrastructure Overview

Full module map, ECS cluster, and the jumphost-centric ingress model.

ECS Fargate Services

The internal services nginx proxies to, and their Cloud Map names.

VPN IP Auth & Jumphost Routing

The gateway-side of the 9090 header trust boundary.