VPC, WireGuard Jumphost & nginx Routing

Almost every byte of external traffic into SkyHub production passes through one t4g.nano EC2 instance. It is simultaneously the WireGuard VPN server, the nginx TLS reverse proxy, the drone rosbridge jumphost, and the bridge to the on-prem Docker host. It is the architectural keystone of the whole platform — and its single largest point of failure.

There is no ALB / ELB and no Cloudflare tunnel in production ingress. The skyhub-prod-skyhub-wireguard-server instance (public EIP <prod-ingress-ip>, private <jumphost-private-ip>) is the sole public entrypoint. If it is lost, the UI, REST/Socket.IO API, video signaling, gamepad control, drone rosbridge, remote Docker, and OTLP export all go dark at once. It is backed up only by a daily EC2 AWS Backup plan (tag Backup=daily).

The VPC

Defined in environments/prod/vpc_configuration.tf via terraform-aws-modules/vpc/aws v5.15.0. It is deliberately minimal — one AZ, one NAT — with the multi-AZ / database-subnet lines commented out.

Property	Value
CIDR	`172.31.0.0/16`
AZs	1 (`azs = [names[0]]`)
Public subnet	`<vpc-subnet>/20`
Private subnet	`172.31.0.0/20`
NAT gateway	1 (`single_nat_gateway = true`)
AWS VPC resolver	`172.31.0.2` (used by nginx `resolver` to look up Cloud Map names)
Service discovery	Cloud Map private DNS `skyhub-prod.internal`

All ECS Fargate services (gateway, janus, whip, ws_proxy, redis, database) run in the private subnet and are reachable only through their Cloud Map A-records (for example gateway.skyhub-prod.internal:5000). The jumphost is the one thing in the public subnet. A single S3 Gateway VPC endpoint (modules/vpc_endpoints) offloads S3 traffic from the lone NAT gateway; ECR interface endpoints are commented out.

This is not an HA topology: single AZ, single NAT, single jumphost, and every ECS service runs desired_count = 1 on FARGATE_SPOT. See ECS Fargate Services and AWS Infrastructure Overview for the service inventory and the reasoning.

The WireGuard jumphost instance

Provisioned in modules/wireguard/wireguard.tf.

Attribute	Value
Instance type	`t4g.nano` (ARM Graviton)
AMI	`ami-0669c5d030c83a310` (Ubuntu)
Root volume	25 GB gp3
Public ENI	in public subnet, has Elastic IP `<prod-ingress-ip>` → Cloud Map `jumphost-public.skyhub-prod.internal`
Private ENI	in private subnet (`<jumphost-private-ip>`) → Cloud Map `jumphost-private.skyhub-prod.internal`
SSH	TCP 3377 (not 22), key from SSM `/skyhub-prod/wireguard_ssh_key`
WireGuard	UDP 51820 (server `10.69.0.1/16`)

The dual-ENI design matters: internal callers use jumphost-private... (the private ENI, which stays inside the VPC and never egresses through the NAT), while browsers/drones use the public EIP. The gateway env deliberately points remote-Docker, VPN, OTLP, and rosbridge routing at jumphost-private.skyhub-prod.internal for exactly this reason.

WireGuard address plan

The instance runs the WireGuard server that stitches AWS to the on-prem office network and to physical drones. Three /16 planes are carved out (wireguard.tf variables):

Plane	CIDR	Purpose
Infra / server	`10.69.0.0/16`	Jumphost (`10.69.0.1`/`.8`), office Docker host, SigNoz collector
Users	`10.70.0.0/16`	Per-user VPN clients (see User VPN & Network Isolation)
Drones	`10.71.0.0/16`	Physical drones; iptables rules per `user_drone_access` scope reachability

The office/on-prem Docker host is addressed everywhere as <office-docker-host>, but in the static wg_peers list that address is labelled spare-key-1 — the real nexus0 host reuses that peer address. Do not “clean up” that peer without understanding it is load-bearing.

The nginx routing table

This is the heart of the page. nginx on the jumphost TLS-terminates the public ports (Let’s Encrypt certs at /cert/fullchain.pem + /cert/privkey.pem, issued by certbot for the frontend domain) and reverse-proxies each listen port to an internal Cloud Map service or an over-VPN host. The authoritative reference is nginx/default; the same config is regenerated at instance boot by environments/prod/wireguard/scripts/post_install_1.tftpl.

Listen (jumphost)	TLS	Upstream	Purpose
`80` / `443`	yes	`skyhub-prod-ui-bucket.s3-website.eu-central-1...`	Static Angular SPA (proxied to the S3 website endpoint)
`5000` `/socket.io/`	yes	`gateway.skyhub-prod.internal:5000`	Socket.IO telemetry (WebSocket-upgrade headers set)
`5000` `/`	yes	`gateway.skyhub-prod.internal:5000`	REST API — auth, drones, missions, billing, assets
`7070`	yes	`ws_proxy.skyhub-prod.internal:7070`	Gamepad WebSocket (WS Proxy)
`8188`	yes	`janus.skyhub-prod.internal:8188`	Janus WebRTC signaling (Janus)
`9090`	no	`http://$http_x_drone_ip:$http_x_drone_port`	Dynamic drone rosbridge passthrough (see below)
`2375`	no	`<office-docker-host>:2375`	Remote Docker Engine API on the office host (SITL)
`4317`	no	`<office-docker-host>:4318`	OTLP traces → SigNoz collector on the office host
UDP `20000-21100`	—	`janus...:$server_port`	WebRTC media relay (nginx `stream {}` block)

The Cloud Map upstream locations (5000/7070/8188) each use resolver 172.31.0.2 valid=10s with the host held in a set $upstream_endpoint variable, so nginx re-resolves the Cloud Map A-record on a 10 s TTL rather than pinning an IP at reload — essential because Fargate tasks get new private IPs on every redeploy. Other blocks differ: the 80/443 S3-website block instead uses resolver 8.8.8.8 (public DNS) with the bucket host hardcoded in proxy_pass, 9090 uses neither a resolver nor a set-variable (it proxies to $http_x_drone_ip), and 4317 proxies to a static <office-docker-host> with no resolver.

The `x-drone-ip` / `x-drone-port` jumphost trick

Port 9090 is a dynamic reverse proxy: nginx forwards to whatever host and port the request declares in its headers, with proxy_pass http://$http_x_drone_ip:$http_x_drone_port; (nginx/default:160). There is no upstream list — the client picks the destination. The gateway is the client. In src/rosbridge/connection.py, when JUMPHOST_IP is set (prod: jumphost-private.skyhub-prod.internal, JUMPHOST_PORT=9090), the Connection object dials the jumphost instead of the drone and attaches the target as headers:

src/rosbridge/connection.py

self.direct = not self.jumphost_ip
...
if not self.direct:
    self.url = f"ws://{self.jumphost_ip}:{self.jumphost_port}"
...
# per-connection websocket handshake (connection.py:162)
header={} if self.direct else {"x-drone-ip": str(self.drone_ip), "x-drone-port": str(self.drone_port)},

If JUMPHOST_IP is empty (local/dev), direct = True and the gateway opens ws://<drone_ip>:9090 straight to the drone with no headers. This one flag makes SITL, physical, and jump-routed drones interchangeable from the connection layer’s perspective.

Port 9090 is effectively an open, unvalidated proxy (an SSRF primitive): nginx will connect to any x-drone-ip/x-drone-port a client supplies. It is safe only because it lives behind the WireGuard network and security group. Never expose 9090 more widely, and never add validation-free header proxying on a new port. The gateway-side counterpart of this trust boundary is documented in VPN IP Authentication & Jumphost Routing.

Two separate paths to the office server

The on-prem office host (nexus0 @ <office-docker-host>, reached over WireGuard) is contacted for two unrelated purposes over two different transports. A refactor must preserve this split:

SITL container management

Gateway env REMOTE_DOCKER_HOST / DOCKER_HOST = tcp://jumphost-private.skyhub-prod.internal:2375. nginx :2375 forwards the Docker Engine API to <office-docker-host>:2375. Used to spawn/stop SITL container stacks — see SITL Drone Lifecycle.

Rosbridge telemetry

Gateway env DOCKER_HOST_IP = <office-docker-host>. For SITL drones the gateway opens the rosbridge WebSocket directly to <office-docker-host>:<port> over WireGuard (not via the :9090 header proxy). See Rosbridge Connection & Reconnect.

SSH to nexus0 (REMOTE_DOCKER_SSH_TARGET=ssh://nexus0@<office-docker-host>) uses the three SSH secrets injected from SSM (/skyhub-prod/ssh/...). OTLP traces take a third path: gateway → jumphost-private:4317 → nginx → <office-docker-host>:4318 (SigNoz), covered in OpenTelemetry & SigNoz.

Security group exposure

The jumphost SG (modules/wireguard/wireguard.tf) governs what is reachable and from where. Note which ports are open to the world versus VPC-only:

Port(s)	Proto	Source	Notes
`3377`	TCP	`0.0.0.0/0`	Custom SSH
`51820`, `51822-51824`	UDP	`0.0.0.0/0`	WireGuard server + client tunnels
`80`, `443`	TCP	`0.0.0.0/0`	UI / TLS
`5000`	TCP	`0.0.0.0/0`	REST + Socket.IO
`7070`	TCP	`0.0.0.0/0`	Gamepad WS
`8188`	TCP	`0.0.0.0/0`	Janus signaling
`9090`	TCP	`0.0.0.0/0`	Rosbridge passthrough
`7766`	TCP	`0.0.0.0/0`	Reserved/aux
`10000-21000`	UDP	`0.0.0.0/0`	WebRTC media
`2375`	TCP	VPC CIDR only	Docker API (VPC-restricted)
`8088`	TCP	VPC CIDR only	Janus HTTP API
`5050`	TCP	VPC CIDR only	VPN management service (`VPN_SERVICE_PORT`)
all	all	egress `0.0.0.0/0`	Unrestricted egress

The SG opens UDP 10000-21000 but the nginx stream {} block listens on 20000-21100, so 21001-21100 are configured in nginx yet not admitted by the SG. There is also no 4317 ingress rule — OTLP works only because the private ENI sits inside the VPC CIDR reachable by the gateway task’s own SG rules, not because 4317 is explicitly opened here.

Config drift & gotchas a future editor must preserve

nginx/default vs post_install_1.tftpl diverge

The checked-in nginx/default (the reference / effectively the running config) contains the 4317 OTLP block but no UDP stream {} media block. The boot template environments/prod/wireguard/scripts/post_install_1.tftpl is the mirror image: it has the UDP media stream but no 4317 block, and it proxies :2375 to <office-host> instead of the prod <office-docker-host>. The live box has clearly been reconciled toward nginx/default; treat that file as source of truth and keep the two in sync when editing routing.

frontend_domain is dev.skyhub.ai in prod

local.frontend_domain is literally "dev.skyhub.ai" in environments/prod/provider_config.tf:69, which is what post_install_1.tftpl feeds to certbot and the server_name. The real prod domain is prod.skyhub.ai (that is what nginx/default and the dashboard’s prod env use). The :2375 server block even carries a stale server_name dev.skyhub.ai. Domain handling is inconsistent between the generated and reference configs.

nginx config is mutable, not immutable

The routing table is (re)written by a remote-exec provisioner (null_resource.configure_nginx) over SSH on port 3377, not baked into an immutable image. Re-running terraform apply can re-trigger it. Changes made by hand on the box will be clobbered unless mirrored into post_install_1.tftpl / nginx/default.

Cloud Map names are load-bearing strings

Upstreams are hardcoded Cloud Map hostnames (gateway/janus/whip/ws_proxy/redis/database/jumphost-private.skyhub-prod.internal). Renaming a service or the namespace simultaneously breaks nginx, the gateway env, and every inter-service call.

Adding a new public port

Open the port in the SG

Add an aws_security_group_rule in modules/wireguard/wireguard.tf (source 0.0.0.0/0 for public, or var.vpc_cidr_block for VPC-only, matching the existing pattern).

Add the nginx server block

Add a matching server { listen <port> ...; } to both nginx/default and environments/prod/wireguard/scripts/post_install_1.tftpl. Use set $upstream_endpoint "<service>.skyhub-prod.internal"; + resolver 172.31.0.2 valid=10s;. Add TLS (ssl_certificate /cert/...) if it is a browser-facing port, and the Upgrade/Connection "upgrade" headers if it carries WebSockets.

Re-provision the jumphost

terraform apply re-runs the remote-exec provisioner, or SSH to ubuntu@<prod-ingress-ip> -p 3377 and reload nginx manually (then backport the change so it survives the next apply).

Point clients at it

Wire the corresponding gateway/dashboard env var to jumphost-private.skyhub-prod.internal:<port> (internal callers) or prod.skyhub.ai:<port> (browsers). See Gateway Environment Variables and AWS Production Configuration.

Network & VPN Topology

The platform-wide WireGuard mesh and trust boundaries.

AWS Infrastructure Overview

Full module map, ECS cluster, and the jumphost-centric ingress model.

ECS Fargate Services

The internal services nginx proxies to, and their Cloud Map names.

VPN IP Auth & Jumphost Routing

The gateway-side of the 9090 header trust boundary.

Deployment & Infrastructure

Ecosystem Services

VPC, WireGuard Jumphost & nginx Routing

The VPC

The WireGuard jumphost instance

WireGuard address plan

The nginx routing table

The `x-drone-ip` / `x-drone-port` jumphost trick

Two separate paths to the office server

SITL container management

Rosbridge telemetry

Security group exposure

Config drift & gotchas a future editor must preserve

Adding a new public port

Network & VPN Topology

AWS Infrastructure Overview

ECS Fargate Services

VPN IP Auth & Jumphost Routing

​The VPC

​The WireGuard jumphost instance

​WireGuard address plan

​The nginx routing table

​The x-drone-ip / x-drone-port jumphost trick

​Two separate paths to the office server

SITL container management

Rosbridge telemetry

​Security group exposure

​Config drift & gotchas a future editor must preserve

​Adding a new public port

​Related pages

Network & VPN Topology

AWS Infrastructure Overview

ECS Fargate Services

VPN IP Auth & Jumphost Routing

The VPC

The WireGuard jumphost instance

WireGuard address plan

The nginx routing table

The `x-drone-ip` / `x-drone-port` jumphost trick

Two separate paths to the office server

Security group exposure

Config drift & gotchas a future editor must preserve

Adding a new public port

Related pages