A deep learning perception framework for farm digital twins: weed species classification, semantic segmentation, and gradient-based visual interpretability
The emergence of Farm Digital Twins (Farm-DT) as a transformative paradigm in smart agriculture demands robust, real-time perception modules capable of continuous plant-level monitoring, predictive analytics, and automated decision support. A critical bottleneck in operationalising Farm-DTs is the absence of interpretable, species-level weed identification engines that can feed spatially precise weed-distribution data into the virtual farm replica for simulation, yield forecasting, and optimized herbicide scheduling. This paper presents a tri-component deep learning framework designed explicitly as a perception and interpretability layer for Farm Digital Twin architectures, evaluated on a five-species balanced subset of the Moving Fields Weed Dataset (MFWD)—a publicly available benchmark of 94K high-resolution images of 28 weed species. Eleven ImageNet pre-trained architectures spanning convolutional neural networks (CNNs) and vision transformers were benchmarked under a unified stratified 80/10/10 holdout protocol. Swin Transformer v1 achieved the highest test accuracy of 97.3% [ F 1 = 0.964; 95% CI: (95.9, 98.4)], and EfficientNetV2-S reached 95.5% [ F 1 =0.956; 95% CI: (93.8, 96.9)]; pairwise McNemar tests confirm these advantages are statistically significant ( p < 0.01, Bonferroni-corrected). Gradient-weighted Class Activation Mapping (Grad-CAM) applied to the top CNN models confirmed that 94.3% of high-activation pixels overlap with ground-truth foliage annotations, providing quantitative validation that classification decisions are driven by botanically meaningful morphological features. The SegFormer-B3 semantic segmentation module achieved mIoU = 0.8961, enabling precise pixel-level weed delineation that directly populates the DT spatial model for variable-rate herbicide application and robotic weeding simulation. Together, these three components map directly onto the Farm-DT architecture: the classifier feeds the species inventory, the segmentation module populates the spatial weed model, and Grad-CAM provides the trust layer required for agronomist acceptance of DT-automated decisions—collectively advancing real-time monitoring and decision-support for Digital Twin-driven urban and peri-urban agriculture.