Leveraging computer vision and natural language processing for efficient metadata extraction in digitized books

Institutional Books - Visual Elements: An open-source pipeline for extracting, classifying, deduplicating, and captioning visual elements from digital book collections

An open-source end-to-end pipeline for detecting, classifying, deduplicating, and captioning visual elements from historical book collections is introduced and an initial dataset of 22.6 million visual elements extracted from the Institutional Books: Harvard Library dataset is released.

Handwritten Text Extraction and Digitization

Extracting structured information from visually rich documents remains a complex task due to variations in layout, text alignment, and reading order. Traditional methods based on IOB tagging or graph decoding often struggle with irregular text sequences and the computational burden of large relational graphs. This paper introduces a novel anchor-based approach that redefines entity representation and association for structured information extraction. The proposed model, named Hwte, integrates visual and linguistic features through a multi-modal transformer architecture that jointly detects entities and their relationships. A new pre-training objective, Masked Detection Modelling (MDM), is introduced to enhance the model’s ability to predict both textual and spatial information simultaneously. Experimental evaluations on benchmark datasets demonstrate that the proposed method achieves superior accuracy and robustness compared to existing solutions, highlighting its effectiveness for real-world document understanding tasks.

Scalable Visual Attribute Recognition in E-Commerce Products via Automated Synthetic Label Generation

Automated Indexing of Historical Postcards: An End-to-End Approach Combining Image and Text Analysis

An end-to-end approach for automated historical postcard indexing that integrates computer vision and natural language processing techniques is presented and effective integration of multiple AI techniques for automated heritage document analysis is demonstrated.

CroQS: Cross-modal Query Suggestion for Text-to-Image Retrieval in Image Archives

The novel task of cross-modal query suggestion is introduced, which interactively guides users by suggesting textual refinements based on visual clusters identified in the retrieval results, and the creation of CroQS, a benchmark dataset comprising 50 diverse queries and 295 semantic clusters in generic domain.

Reproducible Multimodal Artificial Intelligence Workflow for Historical Digital Archive Discovery.

Historical digital archives are increasingly searchable, but discovery remains limited when optical character recognition errors, visually heterogeneous document types, and sparse metadata are handled separately. This study aimed to develop a reproducible protocol to evaluate whether a conservative multimodal artificial intelligence workflow can improve archival retrieval without replacing archivist review. A corpus of 1,600 digitized archival records from eight document classes was assembled, and a 420-query benchmark was used to compare five retrieval conditions: baseline indexing, optical character recognition correction alone, visual classification alone, metadata enrichment alone, and full multimodal integration. Record-level outcomes included character error rate (CER), word error rate (WER), named-entity recall, document-type classification accuracy, prediction confidence, metadata completeness, and subject-heading match. Query-level outcomes included P@10, R@10, nDCG@10, time to first relevant result, and successful-search rate. Optical character recognition correction reduced WER most strongly for handwritten letters, ledgers, and registry books; visual fine-tuning improved classification accuracy most for maps, posters, newspapers, and registry books; and metadata enrichment increased completeness across all historical periods. The full multimodal condition achieved the highest retrieval performance (P@10 = 0.624, R@10 = 0.492, nDCG@10 = 0.634) and reduced the mean time to first relevant result from 156.079 s to 58.485 s. These results support a modular, auditable workflow in which text, image, and descriptive signals are combined under explicit thresholds and human review.