This page covers everything the pilot sees and touches inside the live-flight cockpit: the WebRTC video panel, the gimbal/zoom/record/photo controls, the ArUco precision-landing overlay, and the three ways an operator drives the vehicle (physical gamepad, keyboard, on-screen virtual joysticks) plus the flight-control bar (mode / arm / takeoff / land). All of this lives in HomeModule and is wired through AppStateService.controllerData$.
This page is about the cockpit UI and its command payloads. The lower-level transport mechanics live on sibling pages, and this page links to them rather than repeating them:

Vehicle Commands & Gamepad (redispad)

The VehicleCommandService WebSocket, reconnect, and message routing.

App State & Video (Janus/WebRTC)

JanusService internals and the AppStateService store.

Real-time Telemetry Client

The Socket.IO telemetry stream that drives mode/altitude readouts.

Real-time Transport Channels

Why there are three concurrent channels with opposite directions.

The three channels touching this screen

The cockpit deliberately straddles all three of the Dashboard’s real-time transports. Conflating them is the single most common source of bugs, so keep the directions straight:
ChannelServiceDirectionCarries
Socket.IO telemetryWebsocketServicevehicle → UImode, altitude, GPS (drives the flight-control readouts)
redispad WebSocketVehicleCommandServiceUI → vehiclegamepad axes, buttons, gimbal, camera_command, guided_control, quick set_mode: LAND
redispad WebSocket (2nd socket)ArucoOverlayServicevehicle → UIaruco_tracking frames for the landing overlay
WebRTC mediaJanusServicevehicle → UIthe H264 video tracks
VehicleCommandService and ArucoOverlayService open two separate WebSockets to the same redispad proxy — one for commands, one for tracking. A refactor that tries to “share one socket” will silently drop the ArUco overlay. Both connect to ${environment.ws_proxy}/redispad/{droneId}?access_token=<JWT> (vehicle-command.service.ts:63, aruco-overlay.service.ts:89).

Live video — the Janus subscriber lifecycle

VideoWindowComponent (src/app/home/video-window/video-window.component.ts) owns the <video> element and drives JanusService (src/app/services/janus.service.ts). The room comes straight off the selected drone: roomId = drone.video_room_id, pin = drone.video_room_password — both minted server-side when the vehicle’s Janus room is created (see

Janus Video Rooms

Gateway video rooms
).
1

Drone selected → schedule start

selectedDrone$ fires (deduped by drone id). The component disconnects any prior session, then scheduleVideoStart(roomId, password, 500) debounces a start so a stale timer can’t open a session for the previous drone (video-window.component.ts:246).
2

Init + attach as publisher

onStart() calls janusService.initJanus() (Janus debug logging is forced off to prevent memory accumulation), then new Janus({ server: janusGatewayUrl }) and attachVideoRoomPlugin(), which joins the room with ptype: 'publisher' and the pin (janus.service.ts:54).
3

Discover publishers → subscribe

On the joined event, extractStreams() reads msg.publishers and subscribeTo() attaches a second handle as ptype: 'subscriber' (createNewSubscription). This is the standard Janus VideoRoom two-handle pattern.
4

Answer the offer

The subscriber handle receives an SDP offer (jsep); createAnswer() replies and sends { request: 'start' } back to the room (janus.service.ts:112).
5

Attach the track

handleRemoteTrack() accumulates each inbound track into a single MediaStream, calls Janus.attachMediaStream(videoElem, stream), and pushes videoStreamStatus$ = { available: true }, which clears the “Connecting…” spinner (janus.service.ts:408).

Connect states and gotchas

  • No video_room_id → after a 5s grace period the component toasts “Video streaming not configured for this drone” and does nothing else (handleDroneSelection, video-window.component.ts:218).
  • 15s fallback timeout — if no track and no error arrive within 15s, isConnecting is cleared so the Start Video button reappears instead of spinning forever (video-window.component.ts:450).
  • Start Video button calls droneService.startVideoStream(video_room_id)GET /video_room/{id}/start?update=true, then re-schedules a Janus connect after 1s (video-window.component.ts:540). Sibling REST calls: /video_room/{id}/stop and /video_room/{id}/restart.
  • Vehicle stream sync — the vehicle emits video_stream_state { is_streaming } over redispad; the component auto-connects when the vehicle starts streaming and tears down when it stops (handleVideoStreamState, video-window.component.ts:295). A video_stream_status_request is sent 500ms after drone selection to sync initial state.
A dying Janus session must not reload the app. The destroyed callback deliberately just drops the session and resets local flags so Start Video can re-init — reloading would nuke the map, telemetry and re-download every 3D asset. Proactive teardowns pass notifyDestroyed: false so destroyed only fires on unexpected drops (video-window.component.ts:468, destroyJanusSession at :260).

Gimbal, zoom, record & photo

Two very different payload shapes travel the redispad channel from this panel, both via AppStateService.setControllerData(...):
Hold-to-move buttons emit a GamepadData frame with a single button set to 1 every 80ms while held, and a release frame with value 0 on mouseup/mouseleave/touchend. The button ids are a hard contract shared with the physical gamepad:
Button idAction
12Gimbal pitch up
13Gimbal pitch down
14Gimbal yaw left
15Gimbal yaw right
8Gimbal center (also resets zoom to 0%)
src/app/home/video-window/video-window.component.ts
private sendGimbalUp() {
  const ourTimestamp = performance.now();
  const data: GamepadData = {
    buttons: { '12': 1 }, axes: [],
    timestamp: ourTimestamp, front_ts: ourTimestamp,
  };
  this.appStateService.setControllerData(data);
}
Status feedback loop. The vehicle answers on the redispad channel with a camera_status message. ControllerDataSenderService routes it into websocketService.cameraStatusSubject, and VideoWindowComponent reads is_recording, recording_elapsed_seconds and zoom_percentage from it to keep the record timer and zoom slider truthful (controller-data-sender.service.ts:90, video-window.component.ts:120). A local 1s timer increments the displayed record time between backend syncs.
Because gimbal frames are plain GamepadData buttons, they ride the exact same gate as manual flight input — but buttons are never gated (only axes are). So gimbal, photo and center always reach the vehicle even when guided control is off. See the gate below.

ArUco precision-landing overlay

ArucoOverlayService (src/app/services/aruco-overlay.service.ts) opens the second redispad socket, filters for aruco_tracking messages, and draws marker geometry onto a <canvas> layered over the <video> — the tracking is not burned into the video stream, it is a frontend overlay.
  • Composite markers — an outer ArUco tag and an inner AprilTag. The currently-active marker (chosen by altitude via marker_switch_altitude) is drawn green, the other red; corners, a center-to-crosshair dashed line, and an info panel (distance, yaw, confidence, yaw/XY alignment) are rendered.
  • Performance — the socket and its per-frame draw run via ngZone.runOutsideAngular (canvas draws, no template bindings), and a ResizeObserver keeps the canvas bitmap matched to the video’s displayed size. When the video is hidden (0×0), overlayVisible goes false and message processing pauses.
  • Staleness — if no frame arrives for dataStaleTimeoutMs (2000ms, aruco-tracking.model.ts), the overlay is cleared so it never shows a frozen marker.
  • SITL is skippedVideoWindowComponent only calls arucoOverlayService.connect() when drone.type?.toLowerCase() !== 'sitl' (video-window.component.ts:214).
For the on-drone side that produces these frames, see

YOLO Detection, ArUco Landing & Docking

detection & landing
.

Manual flight — three input paths, one gate

Three input sources all converge on AppStateService.controllerData$, are bridged to the vehicle by ControllerDataSenderService, and gated by guided control:
PathSource fileRateNotes
Physical gamepadgamepad.service.ts60Hz (17ms)Polls navigator.getGamepads(); deadzone 0.1; R1/button 5 plays a shutter sound on edge
Keyboardgamepad.service.ts (same service)60HzArrow keys → progressive-acceleration yaw/throttle on axes; AWSD/Q → gimbal buttons only when video is fullscreen; A → GoTo mode when in GUIDED
Virtual joysticksvirtual-joystick.component.ts80msDual on-screen sticks; maps to a 4-element axes array (axes[0..3]) for quad; rover adds throttle button 7
All high-frequency loops run under ngZone.runOutsideAngular and only re-enter the zone for template-bound side effects (e.g. a docking-mode toast). Losing that pattern reintroduces heavy change-detection churn on every frame.

The guided-control gate

ControllerDataSenderService (src/app/services/controller-data-sender.service.ts, wired once from AppComponent.ngOnInit) subscribes to controllerData$ and decides what actually reaches the vehicle:
src/app/services/controller-data-sender.service.ts
const hasAxesMovement =
  data.axes && Array.isArray(data.axes) && data.axes.some((v) => Math.abs(v) > 0.1);

// Only send axes movement data if guided control is enabled.
// Always allow button commands and guided_control commands through.
if (hasAxesMovement && !this.appStateService.isGuidedControlEnabled()) {
  return; // drop the frame
}
this.vehicleCommandService.send(data);
Stick/axes movement is silently dropped unless guided control is enabled on the vehicle. Buttons (gimbal, photo, dock), guided_control, and camera_command always pass. Removing this gate would stream raw stick input to a vehicle that is not in GUIDED — exactly the dangerous case the design forbids. The on-drone side maps these axes to GUIDED velocity setpoints, never RC_CHANNELS_OVERRIDE; see

Guided Velocity Control & Safety Model

safe control
.

Enabling guided control (the handshake)

GuidedControlService (src/app/services/guided-control.service.ts) owns the enable/disable/status handshake over redispad:
  • The Enable Manual Control button in MissionControlComponent opens a real-vs-virtual selection dialog, which is subscription-gated (SubscriptionDialogComponent.hasSubscription). Selecting a mode calls guidedControlService.enable() before activating frontend controls (manual-control-dialog.component.ts).
  • enable()/disable() wait up to 3s for a guided_control_status reply, then optimistically assume success (backwards compatibility).
  • Guided control auto-disables when the vehicle leaves GUIDED mode: currentMode$ is watched and any non-GUIDED mode flips the state off and hides the virtual joystick (guided-control.service.ts:64).

The dock / disconnect button

Gamepad button 0 (the “X”/dock button) is the disconnect-all signal. MissionControlComponent.dock() and disconnectGamepad() synthesize a { '0': 1 } frame; the vehicle side acts on it, and the frontend enters docking mode (the video crosshair turns to the docking color). Any subsequent keyboard/gamepad/virtual input clears docking mode with a toast (gamepad.service.ts:88, :260).

Flight-control bar (mode / arm / takeoff / land)

MissionControlComponent (src/app/home/mission-control/mission-control.component.ts) is the primary flight-control strip. Its mode buttons are vehicle-type specific:
VehicleMode options
roverMANUAL, GUIDED, AUTO, RTL
copter (quad)POSHOLD, GUIDED, AUTO, RTL
  • The active mode is read from telemetry, not from the button press — diagnosticsSubject messages are parsed with websocketService.getMode(level, 'Mode') so the highlighted button reflects the vehicle’s real state.
  • Mode / Arm / Takeoff / Land / Start-Mission all route through confirmation or altitude dialogs held in AppStateService (e.g. setModeConfirmationDialogVisible, setTakeoffAltitudeDialogVisible) rather than firing REST calls directly from this component. The underlying REST actions (/drone/action/{arm|takeoff|set_mode|start_mission}) are documented on

    Drone Management & Control Actions

    drone actions
    .
  • Takeoff vs Land share one button, toggled by isArmedAndFlying() = systemStatus() === 'ACTIVE' && altitude() > 0.5.
  • GoTo (onGoTo) sets controlVehicleMode so the next map click sets a destination — the map-click dispatch lives on

    Map & Live Vehicle Tracking

    map tracking
    .
  • A low-latency quick land also exists on the command channel: VehicleCommandService.land() sends { set_mode: 'LAND' } straight over redispad (vehicle-command.service.ts:145).

Configuration

These come from src/environments/environment.ts and are the knobs most relevant to this screen:
KeyPurposeValue
janusGatewayUrlJanus WebRTC SFU signaling URLwss://prod.skyhub.ai:8188
janusIceServersICE/STUN servers for WebRTC['stun:stun.l.google.com:19302']
ws_proxyredispad base for commands + ArUcowss://prod.skyhub.ai:7070
productionToggles verbose video-stream console loggingfalse in dev
Media (video/audio) does not flow through janusGatewayUrl directly — UDP relays via the jumphost in the 20000–21100 range, and the on-drone GStreamer pipeline publishes to WHIP first. See

WHIP Ingest Server

WHIP
and

Video Streaming (RTSP → WHIP/WebRTC)

on-drone video streaming
.