An Intelligent System for Detecting and Recognizing Bilingual Text in Natural Scene Images Using Swin Transformer and Parseq

Abstract

Scene text detection and recognition in multilingual environments remains challenging, particularly for morphologically complex scripts such as Arabic. This paper presents an end-to-end deep learning system for detecting and recognizing Arabic and English text in natural scene images. The detection pipeline combines a Swin Transformer Tiny backbone pretrained on ImageNet-22K, a Feature Pyramid Network for multi-scale fusion, and a DBNet++ head, achieving an F1-score of 94.52% on a strictly held-out 2,000-image ICDAR 2019 MLT test set. For recognition, PARSeq with permutation language modeling is trained on 656,868 cropped word samples (85/15 split), reaching 89.50% word accuracy and 94.13% character accuracy on an 865-character bilingual charset. The system includes a vertical-projection word-segmentation fallback and a smart RTL/LTR ordering algorithm. Ablations show FPN contributes +4.6 pp and DBNet++ +3.22 pp F1. Comparisons with prior baselines are reported as non-comparable references. The contribution is a fully reproducible bilingual Arabic-English pipeline.