Robust Motion Tracking and Classification of Zebrafish With Deep Learning

Abstract

Zebrafish serve as a key pre-clinical model for disease research. Studying disease progression requires long-term tracking of fish motion, particularly when considering the function of the musculoskeletal apparatus. While vision-based tracking has been studied, current approaches typically require substantial fine tuning and often lack robustness, especially when multiple fish need to be tracked and motion occurs along all three spatial dimensions. We propose a simplified setup and a 3D tracking method using only two cameras and two convolutional neural networks (CNN) to localize and track multiple fish robustly. We separate the localization of fish and their classification as an individual. This allows for simple training data acquisition by placing fish into the basin individually, without the need for manual annotation. We evaluate the approach on sets of two, five, and ten fish resulting in multiple object tracking accuracies (MOTA) of 0.997, 0.994, and 0.966, respectively. The mean MOTA over all scenarios is 0.99 compared to a mean MOTA of 0.89 for a state-of-the-art tracking algorithm (idtracker.ai). Finally, we present comprehensive data on the motion activity of individual fish in a 30-minute video sequence, demonstrating robust tracking and unique activity profiles of each fish.