Every drone the Gateway talks to has exactly one persistent WebSocket to its on-board rosbridge server (port 9090). This page is a deep dive on the two classes that own that socket:
  • Connection (src/rosbridge/connection.py) — the per-drone wrapper: a bounded outgoing queue, a consumer thread, subscription tracking/replay, the inbound on_message → Socket.IO callback bridge, and a synchronous service-call RPC.
  • SmartSocket (src/rosbridge/socket.py) — a websocket-client WebSocketApp subclass with a hand-rolled reconnect loop, exponential-ish backoff, give-up detection (is_expired), and ping/pong deadline enforcement.
The connection pool (drone_mapping), get_client / _get_client, ownership checks, SITL-vs-physical IP resolution, and the startup wiring that attaches the Socket.IO emit callback live one level up in DroneControlService. See Connection Pool & Startup Wiring. How commands are built and dispatched on top of a Connection is covered in DroneControlService & Rosbridge Dispatch. The browser-facing Socket.IO side (rooms, telemetry_data, yaw) is Socket.IO Telemetry Streaming.

Where a Connection sits

Anatomy of a Connection

The constructor (connection.py:22) captures config from settings and decides direct vs jumphost routing based on whether JUMPHOST_IP is set:
src/rosbridge/connection.py
self.direct = not self.jumphost_ip
...
if not self.direct:
    self.url = f"ws://{self.jumphost_ip}:{self.jumphost_port}"   # + x-drone-ip / x-drone-port headers
else:
    self.url = f"ws://{self.drone_ip}:{self.drone_port}"
When routing through a jumphost, the target drone is carried in x-drone-ip / x-drone-port request headers (connect_to_rosbridge, connection.py:162); nginx on the jumphost reads them and proxies the upgrade. See VPN IP Authentication & Jumphost Routing. Key instance state:
FieldTypePurpose
outgoing_queuequeue.Queue(maxsize=SEND_QUEUE_SIZE)Bounded backlog of outbound rosbridge frames
consumer_threadthreading.ThreadRuns send_buffered_messages() — the queue drainer
subscriptionsset[str]Raw subscribe frames, replayed on every (re)connect
queuesdict[topic -> list]Rolling buffer of the most-recent inbound messages per topic
pending_service_callsdict[id -> threading.Event]Wakes a blocked RPC caller
service_responsesdict[id -> dict]Stores the matching service_response frame
wsSmartSocketThe actual socket (created lazily in connect_to_rosbridge)
is_liveboolSet by on_open / cleared by on_close

Outbound path: bounded queue + consumer thread

Callers never touch the socket directly. send_message enqueues a JSON frame with a 1-second put timeout:
src/rosbridge/connection.py
def send_message(self, message):
    self.outgoing_queue.put(message, timeout=1.0)   # raises queue.Full after 1s
The consumer thread (send_buffered_messages, connection.py:61) pops frames with a 1-second get timeout, records subscriptions, and sends:
src/rosbridge/connection.py
while True and not self.time_to_join:
    try:
        message = self.outgoing_queue.get(timeout=1.0)
        topic = message.lower()
        if "subscri" in topic and "unsubscri" not in topic:
            self.subscriptions.add(message)       # track for replay (add-only!)
        try:
            self.ws.send(message)
        except Exception:
            if self.requeue:                      # REQUEUE env, default false
                self.outgoing_queue.put_nowait(message)
                time.sleep(0.1)
    except queue.Empty:
        pass   # wake every 1s to re-check time_to_join
A full queue drops commands. If the drone is unreachable and the consumer can’t drain, outgoing_queue fills to SEND_QUEUE_SIZE (default 30). New send_message calls then raise queue.Full after 1s. Raise SEND_QUEUE_SIZE for chatty subscription bursts, but the real fix for a wedged drone is reconnect, not a bigger queue.REQUEUE is off by default. With REQUEUE=false, a frame whose ws.send() throws is logged and discarded. Turning it on re-enqueues the frame (and sleeps 100ms), which can reorder or duplicate sends — enable only if you understand that trade-off.

Subscription tracking & replay

Every subscribe frame that flows through the consumer is added to the subscriptions set. On each successful (re)connect, on_open (connection.py:135) replays the whole set so telemetry resumes automatically after a drop:
src/rosbridge/connection.py
def on_open(self, ws):
    self.is_live = True
    if not self.consumer_thread.is_alive():
        self.consumer_thread = threading.Thread(target=self.send_buffered_messages)
        self.consumer_thread.start()
    if self.subscriptions:
        for subscription in self.subscriptions:
            self.send_message(subscription)   # re-subscribe
    self.last_connect = time.time()
The subscription set is add-only. send_buffered_messages adds every subscribe frame but never removes on unsubscribe. After a reconnect, on_open replays all historically-subscribed topics — including streams a client had explicitly unsubscribed from. A future editor changing unsubscribe behavior must also prune self.subscriptions, or the topic will silently come back on the next reconnect.
Because subscribe frames are keyed only by their string content, and the throttle rate is baked into the frame, the same topic subscribed at two different throttle_rates counts as two distinct set entries. Default throttle is TELEMETRY_THROTTLE_RATE (200ms), except /diagnostics, which is subscribed unthrottled (throttle_rate=0). Frame construction lives in src/rosbridge/request_format.py (get_subscribe_msg).
The get_subscribe_msg docstring says the default throttle is 100ms, but the actual settings.TELEMETRY_THROTTLE_RATE default is 200. Trust the setting.

Inbound path: on_message

on_message (connection.py:89) parses each frame with orjson and branches on op: op == "publish" — buffer the payload and fire the Socket.IO callback:
src/rosbridge/connection.py
if self.queues.get(topic) and len(self.queues.get(topic)) <= 10:
    self.queues[topic].append(msg_data)
else:
    self.queues[topic] = []
    self.queues[topic].append(msg_data)

if self.socketio_callback:
    self.socketio_callback(topic, msg_data)   # -> DroneControlService._emit_telemetry
queues[topic] is not a ring buffer. It grows to 11 entries, then the entire list is discarded and restarts at length 1. [-1] is always the latest sample, which is all the yaw/altitude readers use (_emit_dashboard_data, _monitor_takeoff_and_switch_to_auto), so brief history loss is harmless — but any consumer expecting continuous history will be surprised.The GPS yaw enrichment in _emit_dashboard_data mutates the buffered dict in place, so the queued GPS entry also gains a yaw key. Details in Socket.IO Telemetry Streaming.
op == "service_response" — resolve a waiting RPC (below).

Synchronous service-call RPC

Most drone commands are ROS service calls (arm, set_mode, takeoff, param get/set, mission/geofence push). Fire-and-forget frames go straight through send_message; anything that needs the drone’s answer uses send_service_call_with_response (connection.py:236), which blocks the calling thread on a threading.Event: The contract:
src/rosbridge/connection.py
event = threading.Event()
self.pending_service_calls[service_id] = event
self.send_message(message)
if event.wait(timeout):
    response = self.service_responses.pop(service_id, None)
    self.pending_service_calls.pop(service_id, None)
    if response:
        return {"success": response.get("result", False),
                "values": response.get("values", {}),
                "response": response}
    return {"success": False, "error": "Empty response"}
# timed out
return {"success": False, "error": f"Timeout after {timeout}s"}
on_message’s service_response branch pops the pending event and sets it, keyed by the frame id. Timeouts vary by caller — 5s for param set/get, 10s default, and settings.ROSBRIDGE_SERVICE_TIMEOUT (30s) for mission/geofence push over the WireGuard VPN (drone_control_service.py:839+).
Service-call id collisions. Nearly every frame is built with the hardcoded id "1" (request_format.get_call_service_msg("1", ...)). Because pending_service_calls / service_responses are keyed by that id, two concurrent service calls on the same Connection overwrite each other’s Event and response. It works today only because callers are sequential and each blocks on its own response before the next is issued. Anyone adding parallel service calls on one drone must give each frame a unique id first.
If a service call hangs, check the outbound path before the drone: a full outgoing_queue delays the ws.send, and the RPC timer (event.wait) is already running. The returned {"success": False, "error": "Timeout after Ns"} means either the drone never answered or the answer arrived under a different id (collision). Grep the logs for Service response FAILED for id 1 and Service call 1 timed out.

SmartSocket: probe, connect, keepalive

Bringing a connection up is a two-step in start_connection (connection.py:197): an HTTP reachability probe, then a background thread running the WebSocket event loop.
1

_test_connection (reachability probe)

A plain requests.get to http://<target> with connect+read timeout DRONE_REACHABILITY_TIMEOUT (default 2s). rosbridge rejects non-WebSocket requests with HTTP 400, which is treated as success (“reachable”). Any other status or a connection error raises and aborts the connect. Through a jumphost the same x-drone-ip / x-drone-port headers are attached.
2

connect_to_rosbridge (SmartSocket.run_forever)

Builds the SmartSocket and calls run_forever with explicit TCP keepalive and WS ping options (connection.py:173):
  • SO_KEEPALIVE=1, TCP_KEEPIDLE=3, TCP_KEEPINTVL=3, TCP_KEEPCNT=2, TCP_NODELAY=1 — a dead TCP peer is detected in ~3s idle + 2×3s probes ≈ 9s, with Nagle disabled for low latency.
  • ping_interval=3, ping_timeout=2, reconnect=3, connect_timeout=2 — a WS ping every 3s, pong required within 2s.
SmartSocket.check() (socket.py:288) enforces the ping deadline on each read loop and raises WebSocketTimeoutException("ping/pong timed out") when a pong is missing or late — which funnels into the reconnect logic.
Three independent keepalive/ping timers exist — do not conflate them when tuning:
  1. TCP keepalive 3 / 3 / 2 (this socket, OS level).
  2. rosbridge WS ping interval 3 / timeout 2 (this socket, application level).
  3. Socket.IO client heartbeat SOCKETIO_PING_TIMEOUT=60 / SOCKETIO_PING_INTERVAL=25 — that is the browser ↔ Gateway channel, unrelated to the drone socket. See Socket.IO Telemetry Streaming.

Reconnect, backoff & give-up (is_expired)

SmartSocket runs its own reconnect loop instead of websocket-client’s. The reconnect-decision loop lives in the overridden run_forever while-loop (socket.py:147-158): after each socket teardown it asks _should_reconnect(); on retry it fires on_reconnect() and reconnects via setSock, on give-up it sets is_expired = True and tears down. handleDisconnect (socket.py:302) is the per-disconnect entry point invoked from setSock on a failed connect/read. On this deployment the socket is built with the default dispatcher (connect_to_rosbridge passes no dispatcher, so custom_dispatcher=False), so handleDisconnect only flags has_errored and stops the ping thread — its own reconnect branch (socket.py:319) is gated behind custom_dispatcher and is not taken here. Give-up threshold_should_reconnect() (socket.py:333) returns False (→ is_expired=True) when, since the last successful connect, either:
  • reconnection_attempts_since_last_success >= MAX_RECONNECT_ATTEMPTS (default 15), or
  • (only once previously connected) time.time() - last_connect > MAX_RECONNECT_TIME (default 60s).
Backoff_backoff_timeout() (socket.py:347) returns the base reconnect interval (3s), escalating to 10s after 5 attempts. Higher up, _get_client polls is_expired() and restarts a dead connection lazily on the next command:
src/service/drone_control_service.py
if the_connection and the_connection.is_expired():
    the_connection.start_connection()   # re-probe + reconnect
The 20s backoff tier is dead code. _backoff_timeout checks if attempts > 5: 10 before elif attempts > 10: 20. Since > 10 implies > 5, the first branch always wins and the 20s tier is never returned. Effective backoff is 3s then 10s. Preserve the intent (or fix the ordering) if you touch this.RECONNECT is an undefined name. At socket.py:116, self.reconnect = RECONNECT is reached only if run_forever(reconnect=None). connect_to_rosbridge always passes reconnect=3, so the NameError is latent/unreachable today — but don’t call run_forever without an explicit reconnect.

Queue handling across reconnects

on_reconnect() (connection.py:149) recreates the outgoing queue — but only after the first successful connect, deliberately preserving anything buffered while a SITL container is still booting:
src/rosbridge/connection.py
def on_reconnect(self):
    if self.last_connect != 0:            # not on the very first connect
        self.outgoing_queue = self._make_send_queue()   # drops whatever was queued
    self.last_reconnect = time.time()
Reconnect discards queued messages. Any frames sitting in outgoing_queue when a reconnect fires are silently dropped (except on the initial connect). Commands issued during an outage are not guaranteed to survive the reconnect — the UI should re-issue on failure. Subscriptions do survive, because they’re replayed from self.subscriptions in on_open.
Terminal close can hang the consumer thread. The consumer loop only exits when time_to_join is True, and that flag is set only in Connection.close(). On an involuntary give-up (is_expired), teardown fires on_close, which calls self.consumer_thread.join() while time_to_join is still False — so the join can block indefinitely on the callback thread. This is a known sharp edge (connection.py:128); if you refactor teardown, set time_to_join before joining.

Configuration

All values come from src/application/settings.py. Full catalog in Gateway Environment Variables.
Env varDefaultEffect
SEND_QUEUE_SIZE30Max depth of the per-connection outgoing queue
REQUEUEfalseRe-enqueue a frame whose ws.send() failed
DRONE_REACHABILITY_TIMEOUT2Connect+read timeout (s) for the HTTP probe
MAX_RECONNECT_ATTEMPTS15Attempts since last success before is_expired
MAX_RECONNECT_TIME60Seconds since last connect before give-up (once connected)
ROSBRIDGE_SERVICE_TIMEOUT30.0Blocking timeout (s) for mission/geofence service calls
TELEMETRY_THROTTLE_RATE200Default subscribe throttle_rate (ms); diagnostics forced 0
JUMPHOST_IP / JUMPHOST_PORT"" / 9090If IP set, route via jumphost with x-drone-ip/port headers
TCP keepalive (3/3/2), WS ping (3/2), reconnect interval (3), and connect timeout (2) are hardcoded in connect_to_rosbridge (connection.py:173) — they are not env-driven. Change them there if a slow VPN path needs looser deadlines.

Gotchas to preserve

subscriptions is only ever added to. On reconnect, on_open replays every historical subscribe frame, so a topic a client unsubscribed from comes back. Prune the set if you change unsubscribe semantics.
pending_service_calls / service_responses are keyed by the frame id, which is "1" for nearly all calls. Parallel service calls on one Connection clobber each other. Give each concurrent call a unique id.
on_reconnect recreates outgoing_queue (after the first connect). Commands queued during an outage are lost; only subscriptions are replayed.
On an involuntary give-up, on_closeconsumer_thread.join() can block forever because time_to_join is only set by close(). Set the flag before joining in any teardown refactor.
_backoff_timeout’s > 5 branch shadows the > 10 branch (effective backoff 3s → 10s), and RECONNECT at socket.py:116 is undefined but only reached when run_forever(reconnect=None).

Connection Pool & Startup Wiring

drone_mapping, get_client, ownership checks, SITL vs physical IP resolution, and the Socket.IO callback wiring order.

DroneControlService & Rosbridge Dispatch

How commands and telemetry subscriptions are built on top of a Connection.

Socket.IO Telemetry Streaming

The browser side: rooms, telemetry_data, dashboard yaw enrichment, unsubscribe.

VPN & Jumphost Routing

How x-drone-ip / x-drone-port headers reach the drone through the jumphost.