Local Development with SITL & Field Uplink

You can run the entire SkyCore stack — one skyhub service running both halves (MAVROS + rosbridge + modules, and the agent) — on a plain x86 workstation with no Jetson, no flight controller, and no camera. The mechanism is a Docker Compose overlay, docker-compose.local.yml, that stacks on top of the production docker-compose.yml and swaps the hardware-bound pieces for software: an ArduPilot SITL container instead of a real FCU, a local Redis instead of redis.skyhub-prod.internal, a Docker bridge network instead of WireGuard, the TEST camera and a software encoder instead of a SIYI/NVIDIA pipeline, and disabled ArUco/audio/GPIO. This page covers that overlay in detail. It then covers the separate, unrelated topic the same repo owns: the OpenWrt + Quectel 4G LTE router scripts (utils/openwrt/) that give a drone in the field an internet uplink back to the cloud. For how the Gateway Service spins up SITL containers programmatically (the path a real user hits when they click “add SITL drone”), see SITL Drone Lifecycle and the SITL Simulator ecosystem page — that is a distinct flow from the manual overlay below. For the services themselves, start at the SkyCore Drone OS Overview and Microservices & Container Profiles.

There are two ways to get SITL running locally, and they are configured differently:

Gateway-orchestrated — skyhub_gateway_service (ENABLE_SITL=true, DEPLOYMENT_ENVIRONMENT=local) runs the ArduPilot SITL container plus two containers from the same skyhub-app image (SKYHUB_ROLE=core and SKYHUB_ROLE=agent, sitl_drone_service.py:549,712), injecting env vars per sitl/.env.example.sitl. The agent container is deliberately essential=false with its own CPU cap, so an upload-path OOM cannot take MAVROS and rosbridge down. Ports are auto-calculated per container number. Covered in SITL Drone Lifecycle.
Manual overlay — you run docker compose -f docker-compose.yml -f docker-compose.local.yml up -d yourself. This is the developer/off-hardware path documented here.

The two paths inject different MAVLink port values (see Known divergences). Read that section before debugging a dead link.

Prerequisites & quick start

Per sitl/README.md: Docker 20.10+, Compose v2+, ~20 GB free disk (ArduPilot builds from source), 8 GB+ RAM. The overlay’s sitl service builds skyhub-sitl:local from sitl/Dockerfile.sitl, which is FROM skyhub-sitl-base:latest — so build the base image first, then bring the stack up:

Build + run (from repo root)

# 1. Base image: Ubuntu 22.04 + build deps + pymavlink (sitl/Dockerfile.base)
docker build -t skyhub-sitl-base:latest -f sitl/Dockerfile.base sitl/

# 2. SITL image: clones + builds ArduPilot (copter/rover/plane) and
#    mavlink-router from source (sitl/Dockerfile.sitl)
docker build -t skyhub-sitl:local -f sitl/Dockerfile.sitl sitl/

# 3. Bring up redis + sitl + skyhub.
#    skyhub is gated behind a Compose profile, so activate it:
COMPOSE_PROFILES=skyhub \
  docker compose -f docker-compose.yml -f docker-compose.local.yml up -d

# Watch SITL boot (ArduPilot needs ~30s to come up)
docker compose -f docker-compose.yml -f docker-compose.local.yml logs -f sitl

redis and sitl have no Compose profile in the overlay, so they always start. skyhub is gated behind profile skyhub. If you run up without COMPOSE_PROFILES=skyhub you get only Redis and SITL — the application will silently not start. (The README.md quick start omits this.)

Select the simulated vehicle with env vars consumed by the sitl service (docker-compose.local.yml:47-52):

Optional: pick vehicle + spawn point

export VEHICLE_TYPE=ArduCopter          # or Rover / ArduPlane
export HOME_LOCATION=42.1403890,24.7645490,0,0   # lat,lon,alt,heading
export SIM_SPEEDUP=1                     # simulation speed multiplier

What the overlay changes

docker-compose.local.yml is a thin override — it does not redefine the services, it patches them. Everything below is verbatim from that file.

Area	Production (`docker-compose.yml`)	Local overlay
Flight controller	Real FCU on `/dev/ttyACM0 @115200`	`skyhub-sitl` ArduPilot binary
MAVLink router	`mavproxy` (mavp2p)	`mavlink-router` inside the SITL container
Network	`network_mode: host` (WireGuard)	`sitl-vpn` bridge `10.223.0.0/16` (`skyhub`); SITL stays host-net
Redis	`redis.skyhub-prod.internal`	`sitl-redis` at `10.223.0.2:6379`
VPN IP for Redis channels	resolved from `wg0`	`IP_OVERRIDE=10.223.1.1`
Camera	SIYI ZR30/A8 RTSP	`CAMERA_TYPE=TEST`, `CAMERA_IP=` (empty)
Video encoder	NVIDIA `nvv4l2`	`VIDEO_ENCODER=software` (x264)
Audio	ReSpeaker mic	`AUDIO_ENABLED=false`
GPIO charging	Jetson `/dev/gpiochip1`	`BOOT_CHARGING_ENABLED=false`
ArUco landing	env-gated	`ARUCO_LANDING_ENABLED=false`
`privileged`	`true`	`false` (`skyhub`)
Disabled services	—	`mavproxy`, `rtk-ntrip` → `profiles: ["disabled"]`

The two hardware-only services are neutralised by parking them on a Compose profile nobody activates:

docker-compose.local.yml

mavproxy:  { profiles: ["disabled"] }   # SITL handles MAVLink routing
rtk-ntrip: { profiles: ["disabled"] }   # no real GPS

RTK and the ArUco pipeline are the remaining paths you cannot exercise under this overlay — both need real hardware. See RTK NTRIP and e-ArUco Landing. The overlay does not bind-mount source. docker-compose.local.yml defines no volumes for skyhub, so a code change needs a rebuild.

Inside the SITL container

The skyhub-sitl image is deliberately minimal. sitl/Dockerfile.base is Ubuntu 22.04 + build tooling + pymavlink — no ROS. sitl/Dockerfile.sitl then clones ArduPilot (ARDUPILOT_REF=master) and builds arducopter, ardurover, and arduplane for the sitl board, then builds mavlink-router from source. ROS 2 (MAVROS + rosbridge) is not in this container — it lives in the skyhub container, exactly as in production. Three processes run under supervisord (sitl/supervisord.conf):

Why gcs-pump exists. ArduPilot’s --model json backend will not progress past early boot without a heartbeating GCS peer. MAVROS waits for the autopilot and the autopilot waits for a GCS — a deadlock that leaves /mavros/* services permanently absent. sitl/scripts/gcs_pump.py pumps GCS heartbeats into the router to break it. Its endpoint is the only one configured Mode = Server (127.0.0.1:${GCS_PUMP_PORT}, default 14900 + SITL_INSTANCE); the other three are Mode = Normal, meaning the router sends to them and nothing can initiate a session inbound. The program exits immediately and stays down for non-json models.

sitl.sh (sitl/scripts/sitl.sh) maps VEHICLE_TYPE to a binary + model + default params, then execs ArduPilot with --serial0 tcp:0 so MAVLink is served on TCP 5760 (5760 + SITL_INSTANCE*10 for multi-instance). Vehicle mapping:
VEHICLE_TYPE Binary Default SITL_MODEL Defaults file
ArduCopter / copter arducopter quad copter.parm
ArduRover / rover / Rover ardurover rover rover.parm
ArduPlane / plane arduplane plane plane.parm
mavlink-router.sh (sitl/scripts/mavlink-router.sh) waits for the SITL TCP port, auto-detects the Docker host gateway IP (MAVLINK_HOST_IP override → ip route default → /proc/net/route → 172.17.0.1 fallback), then generates /tmp/mavlink-router.conf and runs mavlink-routerd against it. mavlink-router replaces MAVProxy/mavp2p here (~5 MB vs ~100 MB). See MAVLink Routing (mavp2p) for the production router this substitutes for.

`VEHICLE_TYPE`	Binary	Default `SITL_MODEL`	Defaults file
`ArduCopter` / `copter`	`arducopter`	`quad`	`copter.parm`
`ArduRover` / `rover` / `Rover`	`ardurover`	`rover`	`rover.parm`
`ArduPlane` / `plane`	`arduplane`	`plane`	`plane.parm`

SITL port map

From sitl/README.md and the generated router config:

Port	Proto	Purpose
`5760`	TCP	ArduPilot SITL MAVLink server (`--serial0 tcp:0`)
`9090`	TCP	rosbridge WebSocket (served by the `skyhub` container)
`14600`	UDP	MAVLink → MAVROS (`CORE_MAVLINK_PORT`)
`14777`	UDP	MAVLink → the agent half (`GAMEPAD_MAVLINK_PORT`)
`14551`	UDP	MAVLink → external GCS, e.g. QGroundControl (`GCS_MAVLINK_PORT`)
`14900+instance`	UDP	GCS heartbeat pump → router (`GCS_PUMP_PORT`, `Mode = Server`)
`6379`	TCP	local Redis (`sitl-redis`)

MAVROS SITL tuning lives in sitl/scripts/mavros_params.yaml — notably conn_timeout: 30.0 (raised from 10.0 for Docker/CPU overhead) and timesync_rate: 0.0 (SITL uses synthetic time). Stream rates are raised to 10 Hz by sitl/scripts/set_stream_rates.sh, which waits up to 120 s for /mavros/state to report connected: true before calling /mavros/set_stream_rate — the same bootstrap the production skyhub container performs.

Mock VPN, IP_OVERRIDE & Redis namespacing

In production every Redis channel is prefixed by the drone’s WireGuard wg0 IP (see Redis Message Bus and User VPN). Locally there is no wg0, so the overlay fakes it with a bridge network and a fixed IP: The skyhub service sets IP_OVERRIDE=10.223.1.1 and REDIS_HOST=10.223.0.2. IP_OVERRIDE replaces the wg0 IP lookup, so all IP-namespaced channels become 10.223.1.1:gamepad_input, 10.223.1.1:output, etc.

IP_OVERRIDE is mandatory off-hardware: if the agent half cannot resolve a wg0 IP and IP_OVERRIDE is unset, it raises RuntimeError at boot. The legacy misspelled alias IP_OVRIDE is still honored for backwards compatibility — both must keep working. When the backend/Gateway publishes to this drone it must use the same IP prefix (10.223.1.1), or messages won’t match. Two SITL drones sharing one Redis will also cross-talk on the two global (non-namespaced) channels video_stream_state and video_stream_status_request.

Differences from production (behavior deltas)

Because the overlay swaps hardware for software, several code paths take their non-hardware branch:

TEST camera + software encode

CAMERA_TYPE=TEST makes CameraFactory build a GStreamer test-pattern source; VIDEO_ENCODER=software selects x264enc instead of NVIDIA nvv4l2. The application image bakes ARM/Jetson GStreamer paths, so the SITL env template also overrides GST_PLUGIN_PATH to the x86_64 plugin dir. Detail: Video Streaming.

Video gate = CONNECTED

VIDEO_STREAM_DRONE_STATE=CONNECTED starts the stream as soon as MAVROS connects, rather than waiting for ARMED — convenient for testing since you don’t have to arm to see video.

No GPIO / privileged

privileged: false and devices: [] — BOOT_CHARGING_ENABLED=false guards the Jetson.GPIO import so the charging module no-ops on x86 instead of crashing.

Guided control still works

Manual gamepad → GUIDED velocity control is fully exercisable against SITL (mode/arm/velocity setpoints are all simulated). See Guided Velocity Control & Safety Model and Vehicle Commands.

Known divergences

The local-dev files have drifted, and a future editor must reconcile them against the running stack rather than trust any single file. Verified inconsistencies:

Two config sources for the router disagree. The committed sitl/config/mavlink-router.conf routes to mavros 127.0.0.1:14550, but it is not used at runtime — mavlink-router.sh generates its own /tmp/mavlink-router.conf with endpoints gamepad:14777, gcs:14551, core:14600 (no 14550). Only the generated file is loaded (mavlink-routerd -c /tmp/mavlink-router.conf).
Core FCU port: 14550 vs 14600. docker-compose.local.yml:68 sets FCUURL=udp://:14550@, while the Gateway-orchestrated flow (sitl/.env.example.sitl:106) uses FCUURL=udp://:14600@ — matching the router’s generated core endpoint. Under the manual overlay, MAVROS listening on 14550 will not receive from a router that emits to 14600. Verify the port the running MAVROS actually binds.
Gamepad link direction. The overlay sets MAVLINK=udpout:host.docker.internal:14777 (send-out), whereas .env.example.sitl:34 uses MAVLINK=udpin:0.0.0.0:14777 (listen). Only the udpin form pairs correctly with a router that pushes to :14777.
Host-net SITL vs bridged skyhub. sitl is network_mode: host but skyhub is on the sitl-vpn bridge. The router auto-detects the host’s default gateway, which is not the bridge — reaching the bridged services may require MAVLINK_HOST_IP / DOCKER_HOST_IP tuning.
Stale README.md. sitl/README.md describes the SITL container as running MAVROS + ROSBridge and lists scripts/mavros.sh and scripts/rosbridge.sh — neither file exists, and supervisord.conf runs sitl + gcs-pump + mavlink-router. ROS is served by the skyhub container. Troubleshooting commands there also reference container skyhub-sitl-dev, but the compose container_name is skyhub-sitl.

The Gateway-orchestrated path (SITL Drone Lifecycle) is the internally consistent one (core 14600 ↔ router core:14600, gamepad udpin:14777 ↔ router gamepad:14777); prefer it when you need a known-good link.

Troubleshooting

SITL won't start / port 5760 in use

Check docker exec skyhub-sitl cat /var/log/sitl/sitl.log and sitl_error.log. A stale ArduPilot process holding 5760, or a failed image build, are the usual causes — kill the process or rebuild skyhub-sitl:local. The container healthcheck is pgrep -f "ardu(copter|rover|plane)".

MAVROS not connecting to the FCU

Give SITL 30+ seconds. Confirm mavlink-router connected to SITL’s TCP server (/var/log/sitl/mavlink-router.log, look for the TCP client open), that ROS_DOMAIN_ID matches across containers (default 1, and now genuinely honoured — the MAVROS hardcode is gone), and that RMW_IMPLEMENTATION=rmw_cyclonedds_cpp. Then re-check the FCU port divergence in Known divergences.

No ROS topics visible from the host

DDS discovery needs the same domain: export ROS_DOMAIN_ID=1 && export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp before ros2 topic list.

Redis messages never arrive

Confirm IP_OVERRIDE is set on skyhub and that whatever publishes to the drone uses the same 10.223.1.1 prefix. Remember video_stream_state / video_stream_status_request are global (unprefixed).

Field connectivity — OpenWrt 4G LTE uplink

A drone flown beyond Wi-Fi range still has to reach the cloud (rosbridge to the Gateway, video to WHIP, Redis, presigned asset uploads). SkyCore’s answer in the field is a small OpenWrt router with a cellular modem, joined to the same WireGuard plane as everything else. The management scripts live in utils/openwrt/ and run on the router, not on the Jetson.

Hardware (`utils/openwrt/README.md`)

Component	Detail
Router	HILINK HLK-7628N (MediaTek MT7688, MIPS 24KEc @ 580 MHz, 128 MB RAM, 16 MB flash)
OS	OpenWrt 23.05.5, kernel 5.15, target `ramips/mt76x8` (`mipsel_24kc`)
4G modem	Quectel EG25-GC, QMI interface `/dev/cdc-wdm0`, USB ID `2c7c:0125`, LTE
LAN	`192.168.1.1`, bridge `br-lan` (`eth0.1`, `eth0.2`)
WAN	`wwan0` (4G LTE via the Quectel modem)

utils/openwrt/README.md hard-codes the router login (root / a plaintext password) and a client Wi-Fi SSID/password in the clear. Treat these as secrets — do not copy them into shared docs or commits. Placeholders below use root / <redacted>.

Scripts

Copy the scripts onto the router over SSH, then run them there:

From your workstation

sshpass -p '<redacted>' ssh root@192.168.1.1
# copy scripts up (README also references connect_wifi.sh — see note)
sshpass -p '<redacted>' scp connect_wifi.sh disable_wifi.sh monitor_health.sh root@192.168.1.1:/root/

Script	What it does
`disable_wifi.sh`	Turns the Wi-Fi radio fully off via `uci set wireless.radio0.disabled=1 && wifi down`. Used in flight to avoid the Wi-Fi radio contending with LTE / to cut a power draw.
`monitor_health.sh`	Logs a single health line (uptime, `MemFree`/`MemAvailable`, `nf_conntrack_count`, load, Wi-Fi signal) — meant to be cron’d to spot slow degradation.
`connect_wifi.sh`	Documented as joining an upstream Wi-Fi as a client (SSID `Skyhub`). Not present in the tree — the README references it but only `disable_wifi.sh` and `monitor_health.sh` are committed.

monitor_health.sh is designed to run on a schedule so you can correlate connection drop-outs with resource exhaustion:

On the router — log health every 5 minutes

echo "*/5 * * * * /root/monitor_health.sh >> /tmp/health_log.txt" > /etc/crontabs/root
/etc/init.d/cron restart
cat /tmp/health_log.txt

The README’s degradation playbook: memory trending down → memory leak, reboot; conntrack approaching 15360 → connection table full, reboot; high load → CPU overload. The MT7628 SoC exposes no temperature sensor, so thermal monitoring is not available.

4G signal monitoring

Signal is read straight off the Quectel modem with uqmi against the QMI device:

On the router — cellular diagnostics

uqmi -d /dev/cdc-wdm0 --get-signal-info      # rssi / rsrq / rsrp / snr
uqmi -d /dev/cdc-wdm0 --get-data-status      # connected / disconnected
uqmi -d /dev/cdc-wdm0 --get-serving-system   # carrier / MCC / MNC

Signal-quality reference (from utils/openwrt/README.md):

Metric	Excellent	Good	Fair	Poor
RSSI	> -65 dBm	-65 to -75	-75 to -85	< -85 dBm
RSRP	> -80 dBm	-80 to -90	-90 to -100	< -100 dBm
RSRQ	> -10 dB	-10 to -15	-15 to -20	< -20 dB
SNR	> 20 dB	13 to 20	0 to 13	< 0 dB

The recorded office baseline (2025-11-26, both MIMO antennas connected) is RSSI -73 dBm / RSRP -96 dBm / RSRQ -8 dB / SNR 12.6 dB — a “Good” link. If field signal drops well below these, reposition antennas or the aircraft before blaming the software path.

This LTE uplink carries the WireGuard tunnel that in turn carries rosbridge, video, and Redis. It sits upstream of everything on the Network & VPN Topology page and the User VPN service — a weak cellular link degrades telemetry latency and video before any application-level tuning can help.

​Prerequisites & quick start

​What the overlay changes

​Inside the SITL container

​SITL port map

​Mock VPN, IP_OVERRIDE & Redis namespacing

​Differences from production (behavior deltas)

TEST camera + software encode

Video gate = CONNECTED

No GPIO / privileged

Guided control still works

​Known divergences

​Troubleshooting

​Field connectivity — OpenWrt 4G LTE uplink

​Hardware (utils/openwrt/README.md)

​Scripts

​4G signal monitoring

Prerequisites & quick start

What the overlay changes

Inside the SITL container

SITL port map

Mock VPN, IP_OVERRIDE & Redis namespacing

Differences from production (behavior deltas)

Known divergences

Troubleshooting

Field connectivity — OpenWrt 4G LTE uplink

Hardware (`utils/openwrt/README.md`)

Scripts

4G signal monitoring