9090). This page is a deep dive on the two classes that own that socket:
Connection(src/rosbridge/connection.py) — the per-drone wrapper: a bounded outgoing queue, a consumer thread, subscription tracking/replay, the inboundon_message→ Socket.IO callback bridge, and a synchronous service-call RPC.SmartSocket(src/rosbridge/socket.py) — awebsocket-clientWebSocketAppsubclass with a hand-rolled reconnect loop, exponential-ish backoff, give-up detection (is_expired), and ping/pong deadline enforcement.
The connection pool (
drone_mapping), get_client / _get_client, ownership checks, SITL-vs-physical IP resolution, and the startup wiring that attaches the Socket.IO emit callback live one level up in DroneControlService. See Connection Pool & Startup Wiring. How commands are built and dispatched on top of a Connection is covered in DroneControlService & Rosbridge Dispatch. The browser-facing Socket.IO side (rooms, telemetry_data, yaw) is Socket.IO Telemetry Streaming.Where a Connection sits
Anatomy of a Connection
The constructor (connection.py:22) captures config from settings and decides direct vs jumphost routing based on whether JUMPHOST_IP is set:
src/rosbridge/connection.py
x-drone-ip / x-drone-port request headers (connect_to_rosbridge, connection.py:162); nginx on the jumphost reads them and proxies the upgrade. See VPN IP Authentication & Jumphost Routing.
Key instance state:
| Field | Type | Purpose |
|---|---|---|
outgoing_queue | queue.Queue(maxsize=SEND_QUEUE_SIZE) | Bounded backlog of outbound rosbridge frames |
consumer_thread | threading.Thread | Runs send_buffered_messages() — the queue drainer |
subscriptions | set[str] | Raw subscribe frames, replayed on every (re)connect |
queues | dict[topic -> list] | Rolling buffer of the most-recent inbound messages per topic |
pending_service_calls | dict[id -> threading.Event] | Wakes a blocked RPC caller |
service_responses | dict[id -> dict] | Stores the matching service_response frame |
ws | SmartSocket | The actual socket (created lazily in connect_to_rosbridge) |
is_live | bool | Set by on_open / cleared by on_close |
Outbound path: bounded queue + consumer thread
Callers never touch the socket directly.send_message enqueues a JSON frame with a 1-second put timeout:
src/rosbridge/connection.py
send_buffered_messages, connection.py:61) pops frames with a 1-second get timeout, records subscriptions, and sends:
src/rosbridge/connection.py
Subscription tracking & replay
Everysubscribe frame that flows through the consumer is added to the subscriptions set. On each successful (re)connect, on_open (connection.py:135) replays the whole set so telemetry resumes automatically after a drop:
src/rosbridge/connection.py
throttle_rates counts as two distinct set entries. Default throttle is TELEMETRY_THROTTLE_RATE (200ms), except /diagnostics, which is subscribed unthrottled (throttle_rate=0). Frame construction lives in src/rosbridge/request_format.py (get_subscribe_msg).
The
get_subscribe_msg docstring says the default throttle is 100ms, but the actual settings.TELEMETRY_THROTTLE_RATE default is 200. Trust the setting.Inbound path: on_message
on_message (connection.py:89) parses each frame with orjson and branches on op:
op == "publish" — buffer the payload and fire the Socket.IO callback:
src/rosbridge/connection.py
queues[topic] is not a ring buffer. It grows to 11 entries, then the entire list is discarded and restarts at length 1. [-1] is always the latest sample, which is all the yaw/altitude readers use (_emit_dashboard_data, _monitor_takeoff_and_switch_to_auto), so brief history loss is harmless — but any consumer expecting continuous history will be surprised.The GPS yaw enrichment in _emit_dashboard_data mutates the buffered dict in place, so the queued GPS entry also gains a yaw key. Details in Socket.IO Telemetry Streaming.op == "service_response" — resolve a waiting RPC (below).
Synchronous service-call RPC
Most drone commands are ROS service calls (arm, set_mode, takeoff, param get/set, mission/geofence push). Fire-and-forget frames go straight throughsend_message; anything that needs the drone’s answer uses send_service_call_with_response (connection.py:236), which blocks the calling thread on a threading.Event:
The contract:
src/rosbridge/connection.py
on_message’s service_response branch pops the pending event and sets it, keyed by the frame id. Timeouts vary by caller — 5s for param set/get, 10s default, and settings.ROSBRIDGE_SERVICE_TIMEOUT (30s) for mission/geofence push over the WireGuard VPN (drone_control_service.py:839+).
SmartSocket: probe, connect, keepalive
Bringing a connection up is a two-step instart_connection (connection.py:197): an HTTP reachability probe, then a background thread running the WebSocket event loop.
_test_connection (reachability probe)
A plain
requests.get to http://<target> with connect+read timeout DRONE_REACHABILITY_TIMEOUT (default 2s). rosbridge rejects non-WebSocket requests with HTTP 400, which is treated as success (“reachable”). Any other status or a connection error raises and aborts the connect. Through a jumphost the same x-drone-ip / x-drone-port headers are attached.connect_to_rosbridge (SmartSocket.run_forever)
Builds the
SmartSocket and calls run_forever with explicit TCP keepalive and WS ping options (connection.py:173):SO_KEEPALIVE=1,TCP_KEEPIDLE=3,TCP_KEEPINTVL=3,TCP_KEEPCNT=2,TCP_NODELAY=1— a dead TCP peer is detected in ~3s idle + 2×3s probes ≈ 9s, with Nagle disabled for low latency.ping_interval=3,ping_timeout=2,reconnect=3,connect_timeout=2— a WS ping every 3s, pong required within 2s.
SmartSocket.check() (socket.py:288) enforces the ping deadline on each read loop and raises WebSocketTimeoutException("ping/pong timed out") when a pong is missing or late — which funnels into the reconnect logic.
Three independent keepalive/ping timers exist — do not conflate them when tuning:
- TCP keepalive
3 / 3 / 2(this socket, OS level). - rosbridge WS ping
interval 3 / timeout 2(this socket, application level). - Socket.IO client heartbeat
SOCKETIO_PING_TIMEOUT=60/SOCKETIO_PING_INTERVAL=25— that is the browser ↔ Gateway channel, unrelated to the drone socket. See Socket.IO Telemetry Streaming.
Reconnect, backoff & give-up (is_expired)
SmartSocket runs its own reconnect loop instead of websocket-client’s. The reconnect-decision loop lives in the overridden run_forever while-loop (socket.py:147-158): after each socket teardown it asks _should_reconnect(); on retry it fires on_reconnect() and reconnects via setSock, on give-up it sets is_expired = True and tears down. handleDisconnect (socket.py:302) is the per-disconnect entry point invoked from setSock on a failed connect/read. On this deployment the socket is built with the default dispatcher (connect_to_rosbridge passes no dispatcher, so custom_dispatcher=False), so handleDisconnect only flags has_errored and stops the ping thread — its own reconnect branch (socket.py:319) is gated behind custom_dispatcher and is not taken here.
Give-up threshold — _should_reconnect() (socket.py:333) returns False (→ is_expired=True) when, since the last successful connect, either:
reconnection_attempts_since_last_success >= MAX_RECONNECT_ATTEMPTS(default 15), or- (only once previously connected)
time.time() - last_connect > MAX_RECONNECT_TIME(default 60s).
_backoff_timeout() (socket.py:347) returns the base reconnect interval (3s), escalating to 10s after 5 attempts.
Higher up, _get_client polls is_expired() and restarts a dead connection lazily on the next command:
src/service/drone_control_service.py
Queue handling across reconnects
on_reconnect() (connection.py:149) recreates the outgoing queue — but only after the first successful connect, deliberately preserving anything buffered while a SITL container is still booting:
src/rosbridge/connection.py
Configuration
All values come fromsrc/application/settings.py. Full catalog in Gateway Environment Variables.
| Env var | Default | Effect |
|---|---|---|
SEND_QUEUE_SIZE | 30 | Max depth of the per-connection outgoing queue |
REQUEUE | false | Re-enqueue a frame whose ws.send() failed |
DRONE_REACHABILITY_TIMEOUT | 2 | Connect+read timeout (s) for the HTTP probe |
MAX_RECONNECT_ATTEMPTS | 15 | Attempts since last success before is_expired |
MAX_RECONNECT_TIME | 60 | Seconds since last connect before give-up (once connected) |
ROSBRIDGE_SERVICE_TIMEOUT | 30.0 | Blocking timeout (s) for mission/geofence service calls |
TELEMETRY_THROTTLE_RATE | 200 | Default subscribe throttle_rate (ms); diagnostics forced 0 |
JUMPHOST_IP / JUMPHOST_PORT | "" / 9090 | If IP set, route via jumphost with x-drone-ip/port headers |
TCP keepalive (
3/3/2), WS ping (3/2), reconnect interval (3), and connect timeout (2) are hardcoded in connect_to_rosbridge (connection.py:173) — they are not env-driven. Change them there if a slow VPN path needs looser deadlines.Gotchas to preserve
Add-only subscription set re-enables unsubscribed topics
Add-only subscription set re-enables unsubscribed topics
subscriptions is only ever added to. On reconnect, on_open replays every historical subscribe frame, so a topic a client unsubscribed from comes back. Prune the set if you change unsubscribe semantics.Hardcoded service id '1' → concurrent RPC collision
Hardcoded service id '1' → concurrent RPC collision
pending_service_calls / service_responses are keyed by the frame id, which is "1" for nearly all calls. Parallel service calls on one Connection clobber each other. Give each concurrent call a unique id.Reconnect drops queued outgoing frames
Reconnect drops queued outgoing frames
on_reconnect recreates outgoing_queue (after the first connect). Commands queued during an outage are lost; only subscriptions are replayed.on_close joins the consumer thread without setting time_to_join
on_close joins the consumer thread without setting time_to_join
On an involuntary give-up,
on_close → consumer_thread.join() can block forever because time_to_join is only set by close(). Set the flag before joining in any teardown refactor.Unreachable 20s backoff tier & undefined RECONNECT
Unreachable 20s backoff tier & undefined RECONNECT
_backoff_timeout’s > 5 branch shadows the > 10 branch (effective backoff 3s → 10s), and RECONNECT at socket.py:116 is undefined but only reached when run_forever(reconnect=None).Related pages
Connection Pool & Startup Wiring
drone_mapping, get_client, ownership checks, SITL vs physical IP resolution, and the Socket.IO callback wiring order.DroneControlService & Rosbridge Dispatch
How commands and telemetry subscriptions are built on top of a
Connection.Socket.IO Telemetry Streaming
The browser side: rooms,
telemetry_data, dashboard yaw enrichment, unsubscribe.VPN & Jumphost Routing
How
x-drone-ip / x-drone-port headers reach the drone through the jumphost.
