Two–stage multimodal 3D point localization framework for automatic grape harvesting

作者：Qian Shen, Dayu Xu, Tianyu Guo, Xiaobo Mao, Fang Xia · 发表于：Smart Agricultural Technology · 年份：2025 · DOI：10.1016/j.atech.2025.101062 · 被引用次数：2 · 研究领域：Robotics and Sensor-Based Localization、Smart Agriculture and AI、Image and Object Detection Techniques

This study proposes a lightweight Two–Stage multimodal 3D point localization framework for automated grape harvesting, addressing the challenge of precise 3D harvesting point localization. Unlike traditional methods, it employs a Two–Stage multimodal fusion framework, linking RGB and depth images. In the first–stage, pedicels in RGB images are segmented to generate masks. To tackle missing depth information and outliers, an Adaptive Percentile Filtering and Irregular Group-Based Completion (APF–IGBC) algorithm is proposed, leveraging depth distribution patterns and morphological features of grape pedicels. Guided by the mask, APF–IGBC efficiently filters and complements depth information. In the second stage, semantic features from the mask are integrated into the depth image via the Inward Shrinkage Method (ISM) for pose estimation, extracting three key points on pedicels for precise 3D localization. The framework enhances depth restoration and pose estimation accuracy through multimodal fusion. To address multi-scale pedicel challenges, Shared Self–learning YOLO (SSL-YOLO) is introduced, utilizing a Shared Self–learning Head (SSL–Head) for cross-scale information flow. SSL-YOLO achieves 103.9 FPS (9.8 GFLOPs, 2.7M Params) in instance segmentation and 118.8 FPS (6.1 GFLOPs, 2.6M Params) in pose estimation, demonstrating lightweight efficiency, with AP@50 scores of 99.1% and 99.5%, respectively. In comprehensive experiments on a self-constructed grape dataset, the framework a...