YOLO Detection, ArUco Landing & Docking

SkyCore ships three independent computer-vision autonomy capabilities. They live in two different services, use two different ArUco dictionaries, and each has its own MAVLink path — do not conflate them:

Capability	Where it runs	Vision	Marker / model	Actuation path
e-ArUco precision landing (multirotor)	`core` service module `aruco_landing`	SIYI down-camera RTSP	e-ArUco (`DICT_7X7_250`, outer 228 / inner 11)	Direct pymavlink `SET_POSITION_TARGET_LOCAL_NED` → `mavp2p:14561`
YOLO person/vehicle detection + gimbal tracking	standalone `yolo` service (Isaac image)	camera-proxy `/fast_stream` RTSP	YOLOv11n `yolo11n.pt`	MAVROS `/mavros/mount_control/command` + `/mavros/cmd/command`
RealSense-ArUco rover docking	`isaac-slam` service `docking`	RealSense D435 color+depth	`DICT_ARUCO_ORIGINAL`, marker id 70	Redis `{ip}:gamepad_input` (gamepad actuates rover)

This page covers detection, precision landing, and docking. For the video pipeline the landing camera rides on, see /drone-os/robotics/video-streaming. For the GUIDED-velocity safety model that manual movement (and the landing correction commands) build on, see /drone-os/safe-control. For the Redis channel contract and MAVLink port map, see /drone-os/message-bus and /drone-os/mavlink-topology.

e-ArUco precision landing (multirotor)

The landing module implements Embedded ArUco (Khazetdinov et al., 2021): a large outer marker with a small inner marker printed into its center cell, both from DICT_7X7_250. The outer marker is visible from high altitude; the inner marker gives centimeter precision on final approach. The whole module is off by default and lives entirely in the core service:

File	Responsibility
`docker/core/src/modules/aruco_landing/module.py`	ROS2 module: RTSP capture, altitude/GPS subscriptions, Redis command loop, RTL auto-trigger, gimbal lock, tracking publish (1742 lines)
`docker/core/src/modules/aruco_landing/detector.py`	`EArucoDetector` — detects outer/inner markers, pose (rvec/tvec), yaw, distance
`docker/core/src/modules/aruco_landing/controller.py`	`LandingController` — `LandingState` machine, yaw-align-then-descend, per-zone speeds
`docker/core/src/modules/aruco_landing/drone_controller.py`	`DroneController` — direct pymavlink body-NED velocity, yaw, land, mode

Printing a marker

Generate a print-ready e-ArUco marker with utils/generate_e_aruco_marker.py. The inner marker is always 1/9 of the outer size and is embedded in the outer marker’s center cell:

utils/generate_e_aruco_marker.py

# Single 87 cm marker (matches the shipped EARUCO_OUTER_MARKER_SIZE=0.87)
python3 utils/generate_e_aruco_marker.py --size 87 --outer-id 228 --inner-id 11 --output marker.png

# Large marker tiled across A4 pages (assembles into one big marker)
python3 utils/generate_e_aruco_marker.py --size 60 --tiled --output e_aruco_tiles

# List compatible outer/inner ID pairs for DICT_7X7_250
python3 utils/generate_e_aruco_marker.py --find-pairs

Print at 100% scale (no fit-to-page) and verify the printed dimension with a ruler. The detector estimates distance from the marker’s physical size (EARUCO_OUTER_MARKER_SIZE / EARUCO_INNER_MARKER_SIZE), so a mis-scaled print produces wrong altitude and unsafe descent speeds. The default pair is outer 228 + inner 11 (detector.py:46), chosen because 228 has a black center cell that the inner marker blends into.

Detection and the outer→inner switch

EArucoDetector.detect_both() (detector.py:110) detects both markers each frame; the module’s _detect_with_altitude_switch() (module.py:1292) picks which one to act on based on current altitude. Below EARUCO_SWITCH_ALTITUDE it uses the inner marker, otherwise the outer:

docker/core/src/modules/aruco_landing/module.py:1327

should_use_inner = False
if current_altitude is not None and current_altitude <= self.marker_switch_altitude:
    should_use_inner = True

Pose is estimated with cv2.aruco.estimatePoseSingleMarkers using a camera matrix derived from CAMERA_HFOV; each detection yields center (pixels), center_normalized (−1..1), distance (m), and yaw_degrees (0–360, compass). Frames come from a dedicated RTSP capture thread that opens rtsp://{CAMERA_IP}:{CAMERA_PORT}/{CAMERA_PATH} only while landing is active.

Descent state machine

LandingController (controller.py) drives the descent. It corrects yaw first (no descent during yaw alignment), then descends continuously while nudging XY to keep the marker centered. Speeds are chosen per altitude “zone”, so descent is fast up high and gentle near the ground. The three movement primitives all come from DroneController and bypass MAVROS entirely, talking pymavlink to mavp2p on UDP 14561:

Action	MAVLink message	Frame / notes
XY correction + descent	`SET_POSITION_TARGET_LOCAL_NED`	`MAV_FRAME_BODY_NED`, type-mask `0b0000111111000111` (velocity only). `vx`=forward, `vy`=right, `vz`=down (`drone_controller.py:117`)
Yaw align	`MAV_CMD_CONDITION_YAW`	relative rotation, speed capped at 60°/s (`controller.py:531`)
Touchdown	`MAV_CMD_NAV_LAND`	at `ARUCO_LANDING_DISTANCE` (`drone_controller.py:195`)
Mode change	`MAV_CMD_DO_SET_MODE`	GUIDED with retry before descent (`drone_controller.py:267`)

Landing commands are only sent in GUIDED mode. If the flight mode leaves GUIDED mid-descent, LandingController.on_drone_state() calls abort_landing() (controller.py:170). This lets an operator abort instantly by flipping the transmitter mode switch — a safety property a refactor must preserve.

Auto-trigger on RTL / AUTO

Landing can be started manually (a dock_start command on {ip}:gamepad_input, subscribed in _redis_command_loop, module.py:1398) or automatically during a return-to-launch. When RTL_DOCK_ENABLED, _check_rtl_dock_trigger() (module.py:1481) fires once the drone descends to RTL_DOCK_ALTITUDE and is within RTL_DOCK_HOME_DISTANCE of home (haversine distance from /mavros/home_position/home vs /mavros/global_position/global). The home-distance guard prevents a dock from starting at a mid-mission RTL waypoint far from the pad. It works for both a manual RTL and an AUTO mission whose final leg is an RTL.

Gimbal lock and the canvas overlay

When landing activates, _activate_landing() (module.py:1629) locks the camera: points the gimbal down, sets zoom to minimum (widest view), and triggers autofocus — issued both directly over the SIYI TCP protocol and as a Redis camera_command with source: "aruco_landing" to the gamepad service.

That source: aruco_landing camera command sets gimbal_locked in the gamepad service, so operator gimbal input is silently dropped for the duration of the landing (until dock_stop or disarm). Autofocus is re-triggered at each descent-zone transition because the SIYI ZR30 supports only one-shot autofocus.

Marker tracking is not burned into the video stream (the “CANVAS approach”). Instead _publish_tracking_data() publishes marker position/distance/yaw to Redis {ip}:aruco_tracking at ~10 Hz; the gamepad service forwards it to WS clients and the Dashboard renders it as a canvas overlay on top of the WebRTC video. This keeps the encoded stream clean and saves Jetson GPU — see /dashboard/features/video-and-control.

Configuration

All values are read via the EARUCO_* / ARUCO_* / RTL_DOCK_* env prefixes. The table shows the shipped .env.example values (what actually runs). Note the module’s hard-coded fallback defaults differ where called out — always trust .env.example.

Env var	`.env.example`	Purpose
`ARUCO_LANDING_ENABLED`	`false`	Enable the `aruco_landing` module
`EARUCO_OUTER_MARKER_ID` / `_INNER_MARKER_ID`	`228` / `11`	e-ArUco marker IDs (`DICT_7X7_250`)
`EARUCO_OUTER_MARKER_SIZE` / `_INNER_MARKER_SIZE`	`0.87` / `0.077` m	Physical marker sizes (code fallback `0.45` / `0.05`)
`EARUCO_SWITCH_ALTITUDE`	`2.5` m	Outer→inner switch altitude (code fallback `1.3`)
`RTL_DOCK_ENABLED`	`true`	Auto-trigger landing on RTL/AUTO descent
`RTL_DOCK_ALTITUDE`	`30` m	Altitude that arms the auto-dock (code fallback `15`)
`RTL_DOCK_HOME_DISTANCE`	`15` m	Max distance from home to allow auto-dock
`ARUCO_LANDING_DISTANCE`	`0.5` m	Distance at which `MAV_CMD_NAV_LAND` fires (code fallback `1.0`)
`ARUCO_TARGET_YAW`	`0`	Compass heading to align the airframe to (code fallback `270`)
`ARUCO_LEVEL_0..3`	`20 / 10 / 3 / 1.5` m	Zone altitude thresholds (code fallbacks `10 / 5 / 3 / 1.5`)
`ARUCO_XY_SPEED_LVL0..3`	`0.5 / 0.3 / 0.15 / 0.08` m/s	Per-zone XY correction speed
`ARUCO_DOWN_SPEED_LVL0..3`	`1.0 / 0.5 / 0.3 / 0.15` m/s	Per-zone descent speed
`ARUCO_MAVLINK_CONNECTION`	`udpout:127.0.0.1:14561`	pymavlink endpoint into `mavp2p` (`controller.py:45`)
`CAMERA_HFOV`	`81.0`	Camera horizontal FOV for pose intrinsics
`ARUCO_TEST_MODE`	`false`	Detect/track only, suppress movement commands

The mavp2p udps:14561 endpoint the landing module uses is defined only in the main docker-compose.yml. The minimal docker/docker-compose.installer.yml router omits 14561 (and the RTK port), so precision landing will not send velocity commands under the installer compose.

YOLO person/vehicle detection & gimbal tracking

YOLODetectionService (docker/issac-slam/isaac_ros-dev/src/isaac_ros_common/docker/src/services/detection/yolo_service.py) runs a YOLOv11n model on the camera-proxy /fast_stream RTSP feed and drives the SIYI gimbal to keep a detected subject centered. Its behavior flips on the vehicle’s armed state.

Detection runs as its own service/container, gated by USE_YOLO_SERVICE. It is registered in Isaac’s config (DETECTION_ENABLED) but is not instantiated by isaac-slam/main.py’s _init_services (which starts only slam, tts, audio, mavros_monitor, docking). Do not assume the SLAM orchestrator starts YOLO.

Model: yolo11n.pt, cached at /models/yolo11n.pt (auto-downloaded + ONNX-exported on first run, yolo_service.py:125). Optional GPU via USE_GPU=true.
Source: RTSP_URL (default rtsp://127.0.0.1:8554/fast_stream), sampled at DETECTION_FPS (25). The model runs at conf=0.3, then results are filtered by class.
Classes (COCO): person = class 0; vehicles = [1 bicycle, 2 car, 3 motorcycle, 5 bus, 7 truck]. Confidence gates: PERSON_CONFIDENCE = 0.6, VEHICLE_CONFIDENCE = 0.6.
Tracking: multi-object PersonTrack records (track_quality = 0.7*confidence + 0.3*stability) with MAX_TRACKING_DISTANCE = 100 px, TRACK_TIMEOUT = 2.0 s. A persistence bonus biases the tracker to keep the current target rather than flicker between subjects.

Armed vs disarmed behavior (safety-critical)

The armed branch is a deliberate safety guard: if a vehicle is detected while the drone is armed, the service disarms (yolo_service.py:411, calls gimbal_controller.disarm()), and no person tracking runs. Disarmed, it announces the person over TTS, fires a single photo (MAV_CMD_IMAGE_START_CAPTURE), and tracks.

Gimbal control

The detection GimbalController (gimbal_controller.py) publishes MountControl messages to /mavros/mount_control/command with mode = 2 (MAV_MOUNT_MODE_MAVLINK_TARGETING). It converts the target’s pixel error into yaw/pitch angle deltas using the camera FOV, then clamps to the ArduPilot mount limits. Arm/disarm/photo go through the /mavros/cmd/command service.

Env var	Default	Purpose
`USE_YOLO_SERVICE`	`false`	Start the standalone YOLO detection container
`RTSP_URL`	`rtsp://127.0.0.1:8554/fast_stream`	Detection video source (camera-proxy re-encode)
`PERSON_CONFIDENCE` / `VEHICLE_CONFIDENCE`	`0.6` / `0.6`	Detection thresholds
`DETECTION_FPS`	`25`	Detection loop rate
`CAMERA_HFOV` / `CAMERA_VFOV`	`72` / `42`	Gimbal FOV mapping (SIYI A8)
`MOUNT_YAW_MIN/MAX`, `MOUNT_PITCH_MIN/MAX`	`-135/135`, `-25/90`	ArduPilot `MNT1_*` limits (extra safety clamp at yaw ±120, pitch −20..80)

RealSense-ArUco rover docking

The rover docking system (docker/issac-slam/.../services/docking/) autonomously reverses a ground vehicle onto a charging dock marked with a single ArUco tag. Unlike precision landing, it uses DICT_ARUCO_ORIGINAL (marker id 70), a RealSense D435 for color + depth, and it does not command MAVLink directly — it injects gamepad-format moves onto Redis so the gamepad service actuates the rover.

File	Responsibility
`docker/issac-slam/.../docking/docking_service.py`	`DockingService` — listens on `{ip}:gamepad_input` for the button-0 trigger, debounces, starts the process
`docker/issac-slam/.../docking/docking_process.py`	`DockingProcessService` — D435 capture, ArUco pose, the state machine, movement injection, disarm+charge (1057 lines)
`docker/issac-slam/.../docking/docking_config.py`	`DockingConfig` — all `DOCK_*` tunables

Trigger and state machine

Docking is triggered by gamepad button 0 (Cross) on {ip}:gamepad_input, debounced 1.0 s (docking_service.py) — the only implemented docking trigger. (The AUTO_DOCK_AFTER_MISSION env var is plumbed through docker-compose to the isaac-slam container but is not currently wired to any code path.) The process is a numeric state machine:

SEARCHING rotates in a step-and-pause pattern (rotate DOCK_SEARCH_STEP_DURATION 0.5 s, pause DOCK_SEARCH_PAUSE_DURATION 0.8 s to let detection settle) until marker 70 is found.
APPROACHING reverses toward the marker (throttle = -DOCK_BACKWARD_VALUE) with a proportional steering correction from the horizontal pixel error and marker depth.
REPOSITIONING drives forward a short distance and retries alignment if the rover arrives misaligned (DOCK_MAX_HORIZONTAL_ERROR 0.18).
FINAL_DOCKING applies a final backward push for DOCK_FINAL_DURATION (1.5 s).

Movement injection and finish

Movement commands are published back onto {ip}:gamepad_input in the gamepad JSON format — axes[0] carries steering and throttle is encoded as button "7" (R2, forward) or "6" (L2, backward) — so the existing gamepad control path drives the rover (docking_process.py:789):

docker/issac-slam/.../docking/docking_process.py:806

command = {"axes": [steering] + [0.0] * 9, "buttons": {}}
if throttle > 0:
    command["buttons"]["7"] = throttle   # R2 forward
elif throttle < 0:
    command["buttons"]["6"] = abs(throttle)  # L2 backward

On FINAL_DOCKED, _disarm_and_charge() (docking_process.py:862) waits 10 s, sends a disarm (button "9", toggle-arm), waits another 10 s, then sends a charging_control command to start the on-drone charging relay (Jetson GPIO, handled by the gamepad service). Failure at any state skips disarm/charge and returns a DockingResult(success, duration, final_distance, error_message).

Env var	Default	Purpose
`DOCKING_ENABLED`	`true`	Start the docking service (Isaac orchestrator)
`DOCK_TARGET_MARKER_ID`	`70`	ArUco id to dock to (`DICT_ARUCO_ORIGINAL`)
`DOCK_MARKER_SIZE`	`5.2` cm	Physical marker size for pose
`DOCK_TARGET_DISTANCE`	`0.30` m	Stop distance from marker
`DOCK_MAX_REPOSITION_ATTEMPTS`	`3`	Realignment retries before failing
`DOCK_SAFETY_TIMEOUT` / `DOCK_SEARCH_TIMEOUT`	`60` s / `60` s	Approach + search timeouts
`DOCK_COLOR_WIDTH/HEIGHT`, `DOCK_CAMERA_FPS`	`848x480`, `15`	D435 stream config
`DOCKING_CAMERA_SERIAL`	(unset)	RealSense serial for the docking camera

Gotchas a future editor must preserve

Two ArUco systems, two dictionaries, two paths

Precision landing uses DICT_7X7_250 e-ArUco (outer 228 / inner 11) and commands MAVLink directly over mavp2p:14561. Rover docking uses DICT_ARUCO_ORIGINAL single marker id 70 and actuates via Redis {ip}:gamepad_input. They share no code and no marker set — changing one does not affect the other.

Gimbal is locked during precision landing

aruco_landing sends camera_command with source: aruco_landing, which sets gimbal_locked in the gamepad service. Operator center/move gimbal commands are dropped until dock_stop or disarm. Removing this lock lets operator input fight the landing controller.

YOLO auto-disarm on vehicle-when-armed

When armed, a vehicle detection triggers MAV_CMD_COMPONENT_ARM_DISARM. This is intentional safety behavior, not a bug — keep the armed/disarmed branch split intact.

Landing tracking is a canvas overlay, never burned into video

Marker overlays go to Redis {ip}:aruco_tracking and are drawn on the frontend. The core module deliberately does not composite them into the encoded stream (preserves quality + Jetson GPU). Do not re-add a server-side overlay.

.env.example values override the code fallbacks

Several aruco_landing fallback defaults in module.py (marker sizes 0.45/0.05, switch 1.3 m, RTL altitude 15 m, land distance 1.0 m, target yaw 270°, zones 10/5/3/1.5) differ from the shipped .env.example (0.87/0.077, 2.5, 30, 0.5, 0, 20/10/3/1.5). Production runs the .env.example values — quote those when documenting behavior.

Video Streaming

The RTSP → WHIP/WebRTC pipeline and camera-proxy /fast_stream that YOLO and the canvas overlay build on.

Guided Velocity & Safety

The GUIDED-mode velocity model the landing corrections and docking moves ultimately execute through.

MAVLink Routing (mavp2p)

The single MAVLink hub and the 14550/14561/14777 port map used by these subsystems.

Redis Message Bus

The {ip}:gamepad_input and {ip}:aruco_tracking channel contract.

​e-ArUco precision landing (multirotor)

​Printing a marker

​Detection and the outer→inner switch

​Descent state machine

​Auto-trigger on RTL / AUTO

​Gimbal lock and the canvas overlay

​Configuration

​YOLO person/vehicle detection & gimbal tracking

​Armed vs disarmed behavior (safety-critical)

​Gimbal control

​RealSense-ArUco rover docking

​Trigger and state machine

​Movement injection and finish

​Gotchas a future editor must preserve

​Related pages

Video Streaming

Guided Velocity & Safety

MAVLink Routing (mavp2p)

Redis Message Bus

e-ArUco precision landing (multirotor)

Printing a marker

Detection and the outer→inner switch

Descent state machine

Auto-trigger on RTL / AUTO

Gimbal lock and the canvas overlay

Configuration

YOLO person/vehicle detection & gimbal tracking

Armed vs disarmed behavior (safety-critical)

Gimbal control

RealSense-ArUco rover docking

Trigger and state machine

Movement injection and finish

Gotchas a future editor must preserve

Related pages