Multi-Task Deep Learning with Transformer Encoder-Decoder for Semantic Segmentation

Abstract

Visual understanding is a core element of computer vision and includes image classification, object detection, and segmentation. Through a multilayer process, it gains complex image understanding using deep learning methods for video understanding, implemented by converting images to text. In this study, the system extracts video frames, then processes them using the Transformer and Inception V3 architectures for image captioning. This synchronization is based on the Multitask Deep Learning method, which combines a Convolutional Neural Network (CNN) for the image area, a Recurrent Neural Network (RNN) with LSTM (Long Short-Term Memory) for the sentence area, a Caption Content Network (CCN), and a Relational Network (RCN). Context). We use the Transformer Encoder-Decoder architecture to label objects and model relationships between them. To test image-to-text conversion, we use scores to compare the predicted text with one or more reference translations, along with accuracy and loss validation tables and graphs showing epoch-by-epoch loss. For the image dataset from the CNN features, it will immediately enter multitask learning along with text classification; the score will be calculated, and the result will be image caption text. Future research should examine the generalization gap by assessing both validation loss and training loss. The adoption of techniques such as cross-validation, dropout, or early stopping (e.g., halting training at the precise epoch when validation loss ceases to improve) may be considered to enhance the model's performance on entirely new videos.