Detecting reconciliation discrepancies in tabular data using transformers
Introduction Large organisations maintain heterogeneous datasets in data lakes, where schema variability, inconsistent formats, and semantic ambiguity complicate reconciliation. Achieving a unified view of entities requires methods that capture both structural equality and semantic relationships. Methods We propose a transformer-based methodology that adapts pre-trained language models (PLMs) to tabular data by generating metadata-enriched embeddings (table title, column name, type, and statistics). These embeddings are compared using mutual top-K similarity, value-level verification, and Facebook AI Similarity Search (FAISS) for discrepancy detection. Ground-truth labels were established through manual annotation of 1,000 column pairs per dataset, with three annotators achieving substantial agreement (Cohen’s κ = 0.82). Results Experiments on large-scale tabular data (185,909 tables) demonstrate that semantic embeddings efficiently uncover relationships and discrepancies, achieving precision of 0.958 at τ = 0.9 and F1-scores ranging from 0.77–0.87. Benchmarking against baselines (exact matching, Jaccard similarity, edit distance, TF-IDF/BM25, sentence-transformer embeddings, DeepJoin-style embeddings, and schema-name-only matching) shows that lexical baselines achieve high precision but poor recall, while semantic baselines capture relationships but underperform on heterogeneous tables. In contrast, our metadata-enriched transformer embeddings consistently achieved the highest F1-scores. Discussion Unlike prior schema-aligned or query-driven approaches such as DeepJoin, WarpGate, and Lotus, this study introduces a unified reconciliation pipeline that integrates equality-based and semantic matching for large-scale tabular data. The key contribution is a scalable and generalisable reconciliation methodology that operationalises transformer architectures for automated discrepancy detection in heterogeneous environments, establishing both methodological novelty and practical effectiveness.