Vision-Based Perception Framework for Autonomous Robotic Date Palm Harvesting in Challenging Agricultural Environments

Abstract

Date palm harvesting is a labor-intensive and hazardous agricultural operation due to the height of palm trees, dense canopy structures, and the need for precise cutting of fruiting branches. Enabling autonomous harvesting in such environments requires robust, real-time perception systems capable of operating under occlusion, variable illumination, and embedded hardware constraints. This article presents a two-stage vision-based perception framework designed to support autonomous robotic date palm harvesting. The proposed system combines a lightweight You Only Look Once version 11 nano (YOLO11n) object detector for global localization of date palm fruit bunches with a customized YOLO11n-Pose network for fine-grained estimation of harvest-specific keypoints, including the optimal cutting point on each branch. The cascaded design reflects the operational workflow of a harvesting robot, enabling coarse target detection followed by precise anatomical localization. The framework was evaluated on a custom dataset collected from real Tunisian date palm orchards under challenging field conditions, including 5530 detection images with 24 559 annotated branch instances and a dedicated keypoint subset of 387 images containing 776 annotated branch instances and 3104 labeled keypoints. Experimental results demonstrate. For cutting-point localization, the proposed pose model achieved a test pose mAP@0.5 of 0.860 and a test pose mAP@0.5:0.95 of 0.775. A dedicated cutting-point evaluation yielded end-to-end PCK values of 0.588 at T = 0.10 and 0.676 at T = 0.20, with mean and median image-space localization errors of 41.39and 28.63 px, respectively. Both models were deployed on an NVIDIA Jetson Orin NX embedded platform. In the complete two-stage pipeline, the top-1 ROI mode required 65.16 ms per selected harvesting target, while processing all detected region of interests (ROIs) required 204.95 ms per image.