Deep learning-based motion analysis and object tracking

Abstract

Detecting and tracking moving objects are vital tasks in computer vision, with broad applications in areas like surveillance, self-driving cars, and human-computer interaction. Identifying prominent moving objects in complex and cluttered scenes, especially when facing occlusions and difficult environmental factors such as weather and lighting, is crucial for enabling autonomous systems to perceive dynamic environments accurately. While many deep learning techniques have been developed to tackle these challenges, there remains potential for further advancements. This thesis proposes that integrating traditional methods with deep features provides a promising solution to improve the robustness of deep learning in moving object detection and tracking. The work is organized into three main parts. The first part focuses on motion analysis for single-object and multi-object tracking using the DeepFTSG framework. The second part investigates cross-domain generalization for nuclei and cell segmentation in biomedical imaging. The third part extends the motion-analysis pipeline to multi-object tracking by using deep learning-based detection through Bird- Net, followed by association and trajectory generation using M2Track. A novel deep architecture is proposed, named DeepFTSG, for robust moving object detection that incorporates single and multi-stream multi-channel USE-Net trellis asymmetric encoders extending U-Net with squeeze and excitation (SE) blocks and a single shared decoder network for fusing multiple motion and appearance cues. DeepFTSG is a deep learning based approach that builds upon our previous hand-engineered flux tensor split Gaussian (FTSG) change detection video analysis algorithm which won the CDNet CVPR Change Detection Workshop challenge competition. DeepFTSG generalizes much better than top-performing motion detection deep networks, such as the scene-dependent ensemble-based FgSegNet v2, while using an order of magnitude fewer weights. Short-term motion and longer-term change cues are estimated using general-purpose unsupervised methods – flux tensor and multi-modal background subtraction, respectively. M2Track is a time-efficient, detection-based multiobject tracking system that employs a three-step cascaded data association scheme. It begins with a fast short-term data association based solely on spatial distance, followed by a robust tracklet linking phase that utilizes discriminative object appearance models. Additionally, M2Track features an explicit occlusion handling unit, which accounts for not only the motion patterns of tracked objects but also environmental constraints, such as the presence of potential occluders in the scene, to improve tracking accuracy in challenging conditions. This thesis highlights the significant speedup achieved in M2Track, where the performance boost varies based on the length of the video sequence and the number of detections. On average, the system achieves up to a 300x speedup, dramatically improving the efficiency of multi-object tracking while maintaining accuracy. BirdNet is a deep learning method specifically designed to detect small objects, particularly turkeys and vultures, in infrared (IR) drone video. Following the detection phase, M2Track is employed to filter out false positives, enabling accurate counting of vultures and turkeys in the IR drone footage. The innovations presented in this thesis demonstrate the effectiveness of integrating traditional motion-based methods with deep learning features to enhance the accuracy, robustness, and reliability of motion analysis and object tracking. The proposed methodologies show significant promise within their primary application domains while also offering strong potential for cross-domain adaptation, including nuclei and cell segmentation in biomedical imaging, as well as object detection and tracking in aerial imagery. These contributions provide a foundation for future research toward more automated, precise, and generalizable segmentation and tracking solutions across a broad range of visual analysis applications.