SITLDroneService (src/service/sitl_drone_service.py, ~900 lines) is the heaviest service in the gateway. It turns a single POST /api/v1/drone {type: sitl} request into a running 3-container Docker stack — an ArduPilot SITL simulator, a CORE container (MAVROS + rosbridge + video), and a gamepad container — plus a shared Redis container. Once the stack is up, the gateway connects rosbridge to it and treats it identically to a physical drone: the same DroneControlService command dispatch, the same telemetry emission, the same Janus video room. That interchangeability is the whole point of SITL.
This page documents the container topology, the arithmetic that derives every port from a single container number, the local-vs-remote Docker split (including ECR auth), the readiness waits that make creation a long synchronous operation, the default ArduPilot params applied on boot, and the reverse-order cleanup that runs on any failure.
SITLDroneService is lazily constructed by get_service('sitl') in src/application/app.py and only if ENABLE_SITL=true. Its __init__ then applies a second gate — is_server_environment — so both must pass before a container is ever started. See Service Layer & get_service Factory.Two gates before anything starts
SITL creation is guarded twice. Understanding both saves a lot of “why is nothing happening” debugging.ENABLE_SITL gate (factory level)
get_service('sitl') in src/application/app.py:93 only instantiates SITLDroneService when settings.ENABLE_SITL is true. Otherwise it logs a warning and raises UnknownDroneTypeException.is_server_environment gate (constructor level)
In
__init__ (sitl_drone_service.py:27), if not IS_LOCAL_ENVIRONMENT and not REMOTE_DOCKER_ENABLED, the service sets is_server_environment = True and skips Docker client init entirely. save() and delete() then raise DroneCreationFailed (“SITL drone creation is not supported in server environments”). This is why the production Fargate gateway never spawns SITL — it is neither local nor pointed at a remote Docker host.DEPLOYMENT_ENVIRONMENT=local(local Docker viadocker.from_env()), orREMOTE_DOCKER_ENABLED=true(remote Docker over the office server, with ECR auth).
The container stack
SITL is not one container. As of the CORE refactor, the SITL image no longer runs ROS — CORE does. Four moving parts:| Container | Name pattern | Image (env) | Role |
|---|---|---|---|
| SITL | SKYHUB_SITL_{n}_{name} | SITL_IMAGE_NAME (ardupilot) | ArduPilot SITL binary via supervisord; emits MAVLink, routes to CORE/gamepad/GCS |
| CORE | SKYHUB_CORE_{n}_{name} | CORE_IMAGE_NAME (core:latest) | MAVROS + rosbridge + video streaming + battery SOC. This is what the gateway connects to. |
| Gamepad | SKYHUB_GAMEPAD_{n}_{name} | GAMEPAD_IMAGE_NAME (skyhub-gamepad:latest) | Manual-control / mock-camera FastAPI service |
| Redis | skyhub-redis (preferred) or SKYHUB_SITL_REDIS | SITL_REDIS_IMAGE (redis:7-alpine) | Shared across all SITL instances; gamepad + CORE state bus |
{n} is the container number (1..101); {name} is the user-supplied drone name with spaces replaced by _. Companion names are derived by string substitution in _get_companion_container_names() (sitl_drone_service.py:118) — SKYHUB_SITL_ → SKYHUB_CORE_ / SKYHUB_GAMEPAD_ — so the number and name must stay identical across the three containers or delete/cleanup will miss containers.
Container numbering and port math
find_available_container_number() (sitl_drone_service.py:146) lists all containers (including stopped) whose name starts with SKYHUB_SITL_, parses the number from parts[2], and returns the first free integer starting at 1. It hard-caps at 101 (raises ResourceNotFoundException beyond that). Note the except at the bottom of that method swallows errors and returns 1 — a Docker listing failure silently collapses everyone onto number 1, which then collides.
Every other port is pure arithmetic off that number n. There is no port registry — collisions are prevented only by n being unique.
| Resource | Formula | n=1 | n=2 | Where |
|---|---|---|---|---|
Drone row port / rosbridge | 9090 + n | 9091 | 9092 | save() + start_core_container |
ArduPilot SITL_INSTANCE | n - 1 (TCP 5760 + N*10) | 0 | 1 | start_container |
CORE MAVLink (FCUURL) | 14600 + (n-1) | 14600 | 14601 | start_core_container |
| Gamepad MAVLink | 14777 + (n-1) | 14777 | 14778 | start_container / gamepad |
| GCS MAVLink | 14551 + (n-1) | 14551 | 14552 | start_container |
| Gamepad WS port | 5001 + n | 5002 | 5003 | start_gamepad_container |
ROS_DOMAIN_ID | n | 1 | 2 | start_container / CORE |
The gamepad
ports = {"5001/tcp": gamepad_ws_port} dict is computed but never passed to containers.run — the gamepad container relies on host or shared-namespace networking instead, so the WS port is not published as an explicit Docker port mapping. ROS_DOMAIN_ID = n is what keeps each CORE’s ROS 2 DDS graph isolated from its neighbours.Local vs remote Docker networking
The single biggest branch in this service isuse_remote_docker (REMOTE_DOCKER_ENABLED). It changes the Docker client, the network mode of every container, and how the gateway later reaches rosbridge.
- Local Docker (DEPLOYMENT_ENVIRONMENT=local)
- Remote Docker (REMOTE_DOCKER_ENABLED=true)
- Docker client:
docker.from_env(); no ECR auth (ecr_auth_config = None). - All containers run with
network_mode = "host". rosbridge therefore listens directly on host port9090+n. host.docker.internalis added viaextra_hosts(host-gateway) for CORE/gamepad.- CORE
SKYHUB_SERVER_URL = http://localhost:7080(local WHIP). - The gateway reaches rosbridge at
host.docker.internal:{port}— seeDroneControlService._get_client(drone_control_service.py:565). - CORE and gamepad images must already exist locally —
start_core_container/start_gamepad_containerraiseDroneCreationFailedonImageNotFoundbecause there is no ECR to pull from.
Creation sequence
save() (sitl_drone_service.py:599) is async, long, and synchronous internally (it blocks on healthchecks and readiness probes). The order matters and every failure branch triggers reverse-order cleanup.
1. Quota + numbering
1. Quota + numbering
get_drones_by_type(user_id, 'sitl') is counted against USER_SITL_MAX_COUNT (default 3) → DroneCreationFailed("SITL count exceeded."). Note this is a per-user Docker capacity cap, distinct from the billing limit (can_user_add_vehicle, free tier = 1 SITL) enforced earlier in the route — see Stripe Billing & Vehicle Limits. Then find_available_container_number() picks n.2. Redis
2. Redis
ensure_redis_container() prefers the docker-compose skyhub-redis container, falls back to legacy SKYHUB_SITL_REDIS, and only creates a new redis:7-alpine (port 6379, --requirepass if REDIS_PASSWORD set) if neither exists. It is shared by every SITL instance.3. SITL container + healthcheck
3. SITL container + healthcheck
start_container() validates vehicle_type (rover→Rover, copter/arducopter→ArduCopter), builds the env (see table below), and runs supervisord. Then __check_container_state() polls: if the image declares a Docker healthcheck it waits up to 40 × 2s = 80s for healthy (else DroneCreationFailed); with no healthcheck it just verifies running.4. Gamepad container
4. Gamepad container
start_gamepad_container() starts the manual-control service. IP_OVRIDE (the typo is intentional and load-bearing — it matches the gamepad image config) is set to the SITL container name, used as the drone’s Redis key namespace. MOCK_CAMERA_ENABLED=true generates test images/video. Failure here cleans up the SITL container and aborts.5. CORE container + rosbridge wait
5. CORE container + rosbridge wait
start_core_container() runs MAVROS + rosbridge on 9090+n, then blocks in _wait_for_rosbridge() for up to 30s, GET-ing http://{host}:{port}/ and treating HTTP 400 as “ready” (rosbridge returns 400 to non-WebSocket requests). Timeout only warns — it does not fail creation. Failure of containers.run cleans up SITL + gamepad.6. DB row, video, assets, params
6. DB row, video, assets, params
super().save() persists the Drone with ip == mac == container_name and port = 9090+n, then commits. A Janus room (id == drone.id) + PIN + create_access_token(identity=drone.id) room token are created, update_video_room(..., status="START") auto-starts video (physical drones do not — they defer). Demo assets and _set_default_params follow; both are wrapped in try/except and are non-fatal.Per-container environment
The env dicts are the real contract with each image. Highlights (seestart_container, start_gamepad_container, start_core_container):
- SITL
- CORE
- Gamepad
VEHICLE_TYPE (ArduCopter/Rover), SITL_INSTANCE=n-1, GAMEPAD_MAVLINK_PORT, GCS_MAVLINK_PORT, CORE_MAVLINK_PORT, MAVLINK_HOST_IP=127.0.0.1, WHIP_SERVER_URL, VIDEO_STREAM_DRONE_STATE, DOCKER_HOST. Command: supervisord -c /etc/supervisor/conf.d/supervisord.conf.Default ArduPilot params
After boot,_set_default_params() (sitl_drone_service.py:798) sleeps a fixed 5s (MAVROS param services register after rosbridge is up) and pushes a param batch through DroneControlService.set_params. All values come from SITL_* env vars in src/application/settings.py:
| Param | Source / default | Purpose |
|---|---|---|
LOG_BACKEND_TYPE | 1 (hardcoded) | Enable file logging (required for SITL) |
LOG_FILE_DSRMROT | 1 (hardcoded) | New log per flight on disarm |
FENCE_ENABLE | SITL_FENCE_ENABLE (1) | Geofence on |
FENCE_TYPE | SITL_FENCE_TYPE (7) | Alt + circle + polygon |
FENCE_ACTION | SITL_FENCE_ACTION (1) | RTL on breach |
FENCE_ALT_MAX | SITL_FENCE_ALT_MAX (100) | 100 m ceiling |
FENCE_RADIUS | SITL_FENCE_RADIUS (300) | 300 m circle |
WP_YAW_BEHAVIOR | 3 (hardcoded) | Nose along track |
SIM_BATT_VOLTAGE | SITL_BATT_VOLTAGE (12.587) | Simulated 3S full charge |
BATT_CAPACITY | SITL_BATT_CAPACITY (3300) | mAh |
BATT_LOW_VOLT / BATT_CRT_VOLT / BATT_ARM_VOLT | SITL_BATT_* | Failsafe thresholds |
BATT_FS_LOW_ACT | SITL_BATT_FS_LOW_ACT (2) | RTL on low battery |
set_params routes through the pooled rosbridge connection like any other command — see DroneControlService & Rosbridge Dispatch and Rosbridge Connection & Reconnect.
Deletion and cleanup ordering
Both intentional deletion and failure cleanup stop containers in reverse dependency order: core → gamepad → sitl. Preserve this — CORE holds the rosbridge/DDS session and should go first.delete(drone)(sitl_drone_service.py:755): guardsis_server_environment, deletes all S3 assets (non-fatal), readscontainer_namefromdrone.mac, derives companion names, thenstop_core_container→stop_gamepad_container→ stop/remove the SITL container (30s timeout), and finallysuper().delete()removes the DB row. The shared Redis is left running._cleanup_containers(sitl, gamepad, core)(sitl_drone_service.py:582): the failure path used insidesave(). Each stage ofsave()passes progressively more containers to clean up, so a gamepad failure removes only SITL, a CORE failure removes SITL + gamepad, and a post-start DB failure removes all three.
Gotchas for future editors
ip == mac == container name (not a real IP)
ip == mac == container name (not a real IP)
For SITL, both
Drone.ip and Drone.mac store the container name (e.g. SKYHUB_SITL_1_myDrone), never an IP. DroneControlService._get_client translates it at connect time: remote → SITL_HOST, local → host.docker.internal. Any code that assumes Drone.ip parses as an IP breaks for SITL.find_available_container_number swallows errors and returns 1
find_available_container_number swallows errors and returns 1
The bottom
except returns 1 on any Docker listing error instead of raising, which can force a number collision (and therefore a port collision, since every port derives from n). Watch this if you refactor numbering.Numbering caps at 101, not USER_SITL_MAX_COUNT
Numbering caps at 101, not USER_SITL_MAX_COUNT
The 1..101 range is a global container-number ceiling; the per-user cap is the separate
USER_SITL_MAX_COUNT (default 3). They are different limits.SITL auto-starts video; physical does not
SITL auto-starts video; physical does not
SITL
save() pushes update_video_room(status="START"); PhysicalDroneService.save() leaves the equivalent call commented out (“do this when there is a connection”). This asymmetry is intentional — see Janus Video Rooms & On-Drone Video Control.IP_OVRIDE typo is load-bearing
IP_OVRIDE typo is load-bearing
The gamepad env var is
IP_OVRIDE (missing E). It matches the gamepad image’s own config; “fixing” the spelling on this side without changing the image will break the drone’s Redis namespace resolution. CORE correctly uses IP_OVERRIDE.Related pages
Factory + inheritance: Service Layer & get_service Factory · Command dispatch: DroneControlService & Rosbridge Dispatch · Pool wiring: Connection Pool & Startup Wiring · Env reference: Gateway Environment Variables · The simulator image itself: SITL Simulator and on-drone Microservices & Container Profiles.