DroneControlService keeps exactly one persistent rosbridge WebSocket per drone in an in-memory pool called drone_mapping. Every command dispatch and every telemetry subscription funnels through get_client(user_id, drone_id), which returns the pooled Connection or lazily builds one. This page covers how that pool is keyed and locked, where drone ownership is (and is not) enforced, how SITL and physical drones resolve to different WebSocket targets, and the precise startup order that must run before telemetry can ever flow.
This page is about who gets which connection and when. The
Connection/SmartSocket internals — reconnect, backoff, keepalive, the outgoing queue, and the synchronous service-call RPC — live in Rosbridge Connection & Reconnect. The command methods that consume get_client are in DroneControlService & Rosbridge Dispatch.The pool: drone_mapping
drone_mapping is a plain dict on the DroneControlService singleton, initialized empty in src/service/drone_control_service.py:47:
src/service/drone_control_service.py
| Property | Value |
|---|---|
| Key | drone_id (int) — not (user_id, drone_id) |
| Value | {"connection": Connection} |
| Scope | One Connection per drone, shared across all users and all subscribers of that drone |
| Guard | Single connection_lock (threading.Lock) for the whole pool |
| Eviction | None. Entries persist until close() or an expiry-triggered reconnect. No idle timeout. |
| Lifetime | Process-local — see Multi-worker caveat |
get_client / _get_client
get_client (src/service/drone_control_service.py:518-530) is a thin lock wrapper; all logic lives in _get_client (src/service/drone_control_service.py:532-581).
connection_lock semantics
A single global lock serializes all pool access — not one lock per drone:
src/service/drone_control_service.py
- The lock is held for the whole
_get_clientbody, including theget_drone_by_id_and_userDB query andConnection.start_connection()on a cache miss.start_connectiondoes not block on the WebSocket handshake — that runs on a background thread — but before spawning that thread it runs a synchronous_test_connection()HTTP reachability probe (requests.get, up toDRONE_REACHABILITY_TIMEOUTseconds each for connect and read, default 2s;connection.py:198,218-224). So both that probe and the DB lookup are blocking I/O executed under the lock: a slow DB lookup or an unreachable drone (the probe stalling ~2s+) blocks pooling for every drone, not just this one — and that reachability stall is a more likely trigger ofConnection lock timed outthan a slow DB lookup. - If the lock cannot be acquired within 5 seconds,
get_clientraises. Under heavy concurrent first-connects this surfaces asConnection lock timed out. - The lock protects the dict; it does not protect an individual
Connection’s internal state (its outgoing queue, subscription set, and service-call maps have their own thread model).
Ownership is checked only on creation
Ownership enforcement lives entirely in the cache-miss branch:src/service/drone_control_service.py
get_drone_by_id_and_user(drone_id, user_id) returns None (→ ResourceNotFoundException) when the drone is not owned by that user. But this only runs when the pool misses. If drone 42’s connection already exists — created earlier by its rightful owner — a later get_client(other_user, 42) returns the cached socket without any ownership re-check.
The is_stop_call short-circuit
stop_* telemetry methods (e.g. stop_logs_data) call get_client(user_id, drone_id, is_stop_call=True). When is_stop_call is truthy, _get_client returns whatever is in the pool — possibly None — and never creates a connection:
src/service/drone_control_service.py
SITL vs physical IP resolution
The resolved WebSocket target depends ondrone.type. Physical drones connect to their stored drone.ip; SITL drones store a Docker container name in drone.ip (and drone.mac), which must be translated at connect time (src/service/drone_control_service.py:549-570):
| Drone type | Condition | drone_ip used for rosbridge | Port |
|---|---|---|---|
physical | — | drone.ip (the WireGuard VPN address) | drone.port |
sitl | REMOTE_DOCKER_ENABLED=true | settings.SITL_HOST (default <office-docker-host>, the office server over WireGuard) | drone.port (9090 + n) |
sitl | REMOTE_DOCKER_ENABLED=false | "host.docker.internal" (local Docker host) | drone.port (9090 + n) |
src/service/drone_control_service.py
The
drone.port for a SITL drone is derived arithmetically from its container number (9090 + n); the container name lives in drone.ip/drone.mac. Any code that assumes Drone.ip is a routable IP will break for SITL. The container/port allocation is owned by SITL Drone Lifecycle.SITL_HOST falls back to DOCKER_HOST_IP and defaults to <office-docker-host> (src/application/settings.py:77). Whether the connection then goes direct or through the jumphost is a separate axis handled inside Connection via JUMPHOST_IP/JUMPHOST_PORT and the x-drone-ip/x-drone-port headers — see Rosbridge Connection & Reconnect.
Expiry-triggered reconnect and close
The pool never proactively evicts. Two things end a pooled connection’s life:
- Expiry + reuse. On any
get_client, if the pooledConnection.is_expired()(itsSmartSocketgave up afterMAX_RECONNECT_ATTEMPTS/MAX_RECONNECT_TIME),_get_clientcallsstart_connection()on the same object in place and returns it — thedrone_mappingentry is reused, not replaced. - Explicit
close.close(user_id, drone_id)tears down and removes the entry:
src/service/drone_control_service.py
Startup wiring: the load-bearing order
Telemetry emission is gated onself.socketio being truthy, but the service is constructed without it. Three steps, in this exact order, must complete before any request is served, or telemetry callbacks are silently never attached.
Construct the service without SocketIO (app.py)
src/application/app.py:43 builds the singleton at import time. socketio_instance defaults to None:src/application/app.py
Create the SocketIO instance (main.py)
src/main.py:208 creates the gevent SocketIO server (optionally backed by a Redis message_queue):src/main.py
Why the order is load-bearing
At connection creation,_get_client decides once whether to attach the emit callback, based on the current value of self.socketio:
src/service/drone_control_service.py
Connection construction time and never re-evaluated per message (Connection.on_message just checks if self.socketio_callback). So:
- If a
Connectionis ever created whileself.socketiois stillNone, it is pooled withsocketio_callback=Noneand will stay silent forever, even after step 3 assignssocketio. The only recovery isclose()+ recreate (there is aConnection.set_socketio_callbackmethod, butDroneControlServicenever calls it to retrofit pooled connections). - This is why the comment at
src/main.py:258insists the assignment happens before gunicorn workers start handling requests — noget_clientcall may run beforeself.socketiois set.
Multi-worker caveat
drone_mapping is an in-memory dict on a per-process singleton. When the gateway runs multiple gevent/gunicorn workers, each worker keeps its own pool, so the same drone can hold one rosbridge Connection per worker. SOCKETIO_MESSAGE_QUEUE (an optional Redis URL, src/application/settings.py:82) exists precisely so a telemetry_data emitted from the worker that owns the drone’s rosbridge socket still reaches Socket.IO clients pinned to a different worker. Without it, telemetry only reaches clients that happen to share a worker with the drone’s connection. See Gateway Environment Variables and Startup, Validation & Composition Root.
Things a refactor must preserve
Pool key is drone_id only
Pool key is drone_id only
Changing the key to
(user_id, drone_id) would create one rosbridge socket per user per drone — multiplying load on the drone’s rosbridge server and breaking the “shared telemetry room” model. If you re-key, you must also rethink room naming and ownership.Ownership check is create-time only
Ownership check is create-time only
A cached connection is returned without re-checking ownership. Any per-message authorization must be added explicitly (it does not exist today).
Callback is bound at Connection creation
Callback is bound at Connection creation
socketio_callback is fixed when the Connection is built. Ensure self.socketio is truthy before any get_client, or retrofit via set_socketio_callback — otherwise a connection is permanently mute.Single global lock with a 5s timeout
Single global lock with a 5s timeout
All pool access serializes on one lock. Keep the critical section cheap; do not add blocking I/O inside
_get_client beyond what already runs there — the get_drone_by_id_and_user DB lookup and the synchronous _test_connection() reachability probe inside start_connection() (up to DRONE_REACHABILITY_TIMEOUT, default 2s) — or Connection lock timed out will start firing.SITL ip is a container name, not an IP
SITL ip is a container name, not an IP
Drone.ip/Drone.mac hold the container name for SITL; translation to SITL_HOST or host.docker.internal happens only inside _get_client. Any new code path that opens a socket to a drone must replicate this branch.Related pages
Rosbridge Connection & Reconnect
Connection/SmartSocket internals: outgoing queue, subscription replay, keepalive, backoff,
is_expired, and the synchronous service-call RPC.DroneControlService & Rosbridge Dispatch
The command methods and telemetry emission (
_emit_dashboard_data, yaw calculation) that consume get_client.SITL Drone Lifecycle
Container numbering,
9090 + n port math, local vs remote Docker — the source of the SITL ip/port values this page resolves.Startup, Validation & Composition Root
The full boot sequence in
main.py/app.py that this page’s three wiring steps are part of.
