robot-perception

This skill provides a comprehensive framework for developing and maintaining perception systems in robotics. It is designed for robotics engineers and perception researchers tasked with building robust, high-performance pipelines that integrate diverse hardware such as RGB cameras, structured light depth sensors, LiDARs, and IMUs. Whether you are performing sensor fusion on mobile platforms or implementing visual servoing for robotic manipulators, this skill offers actionable patterns for building reliable perception stacks that function under real-world conditions.

The skill covers the entire lifecycle of perception development, starting from low-level hardware configuration and driver integration to complex algorithm implementation. It emphasizes the importance of accurate geometric calibration and signal synchronization, providing guidance on how to manage multi-sensor rigs, handle perception latency, and ensure frame alignment across distinct modalities. You will find standardized approaches for computer vision tasks including object detection, semantic segmentation, point cloud filtering, and 3D reconstruction.

• Expert guidance on sensor calibration, including intrinsic matrix estimation, extrinsic transformation, and hand-eye calibration protocols for robot-camera systems.

• Comprehensive support for industry-standard tools and frameworks such as OpenCV, Open3D, PCL, and ROS2 perception packages.

• Best practices for managing sensor data, including threaded capture, jitter reduction, and time-stamped synchronization between disparate devices.

• Advanced techniques for point cloud processing, ICP registration, and image undistortion to improve spatial accuracy in navigation and manipulation tasks.

• Production-oriented deployment strategies for edge computing, focusing on GPU acceleration, inference optimization, and handling perception pipeline failures in the field.

• Always prioritize sub-pixel refinement and spatial coverage during checkerboard or charuco board calibration to minimize reprojection errors.

• Implement bounded buffers for sensor streaming to ensure the perception thread never blocks the hardware driver, preventing latency accumulation.

• Use hardware synchronization (e.g., PTP or inter-cam sync cables) whenever possible, falling back to software-based synchronization with strict timestamp windows.

• When debugging misalignment, verify coordinate transforms using tools like tf2 to ensure your frames-of-reference are consistently defined between sensors and the robot base.

• Favor efficient data structures for large point clouds; use PCL filters like VoxelGrid for downsampling before attempting heavy processing or registration.