Analysis of the pre-trained embedding geometry of a small sentence transformer (SBERT) and classic SLM reveals that pre-trained embedding geometry is associated with classification performance and reveals a counterintuitive finding that a structured input that would help a human reader does not improve the SLM performance.
Abstract
Categorising invoices into the correct General Ledger (GL) code underpins financial reporting and tax compliance. This is a skilled accounting judgement rather than a routine task: the correct category depends subtly on the nature of the purchasing business, the vendor and the invoice text. Whilst AI is increasingly being adopted across industries to automate tasks, including invoice categorisation, implementations built on in-house small language models (SLMs) can simultaneously reduce cost and improve data security, confidentiality, and interpretability. We investigate this approach by first analysing the pre-trained embedding geometry of a small sentence transformer (SBERT) and classic SLM (DeBERTa). The sentence-embedding space of this financial corpus is globally anisotropic but composed of locally isotropic clusters, extending prior token-level findings to sentence embeddings in a financial setting, and these clusters are strongly correlated with the vendor identity. SBERT fine-tuned on a single GPU reaches 0.96 accuracy on invoice classification, above both a zero-shot LLM and a vendor identity baseline, increasing performance for smaller, challenging categories and new clients. For this important generalisation problem, SBERT reaches 0.9 F1 with roughly 100 client-specific invoices, showing that an in-house SLM implementation is promising. Combining these results with geometric analysis shows that pre-trained embedding geometry is associated with classification performance and reveals a counterintuitive finding that a structured input that would help a human reader does not improve the SLM performance.
A hybrid detection framework which combines semantically deep embeddings from the RoBERTa transformer with a set of carefully designed language statistics and linguistic statistics and shows excellent resistance to the surface-level adversarial paraphrasing strategy.
Anita Rani, Ms. Suman· International Journal of Sci...· 0 citations
Plagiarism has become an increasingly serious concern, especially within academia, driven by the unlimited availability of online information. It refers to the uncredited use of another author’s text, images, code, graphics, or ideas. There are several types of plagiarism as direct copying, paraphrasing, reformulating, or even translating across languages. Historically, approaches for Cross Language Plagiarism Detection (CLPD) are centered on linguistic and structural cues, multilingual alignment processes, or a measurement of perceived similarity based on concepts to compare. In this study, we propose a new hybrid Approach for CLPD, which combines semantic information from WordNet with the syntactic structure from Universal Dependencies, then these relations are modeled in knowledge graphs for multiple language pairs. In detail, we utilize RotatE for relation embedding, GCN for node embedding, and GAT for improvement in learning contextual representations, to build and embed the graphs. To prove the efficiency of the new hybrid method, three approaches are compared: 1) using only WordNet to build the knowledge graph, 2) using only Universal Dependencies, and 3) the proposed hybrid approach. Our approach demonstrates clear improvements over state-of-the-art baselines across three language pairs: English-Spanish, English-French, English-Arabic, and English-German, on datasets including PAN11, JRC-Acquis, Europarl, Wikipedia, OPUS, and a curated set of conference papers. The proposed hybrid approach achieves Plagdet scores of 98.50% for English-Spanish, 89.02% for English-German, 77.65% for English-Arabic, and 77.67% for English-French.
Chaimaa Bouaine, F. Benabbou, Amine Bouaine et al.· IEEE Access· 0 citations
Public procurement involves the allocation of substantial financial resources; therefore, continuous oversight through audits, controls, and monitoring mechanisms is essential. However, stakeholder comments and publicly available government data are often underutilized, despite their potential to reveal procedural irregularities. To address this gap, this paper analyzes metadata from Ecuador's Sistema Oficial de Contrataci\'on P\'ublica (SOCE, Official Public Procurement System), with particular emphasis on participant comments generated during the pre-contractual phase. We propose a hybrid modeling framework that integrates unsupervised clustering and supervised classification within a natural language processing (NLP) pipeline to uncover latent patterns and detect potentially irregular procurement processes. Semantic embeddings are generated using Word2Vec, LLaMA, and RoBERTa, followed by Gaussian Mixture Models (GMMs) for unsupervised clustering. A supervised classification stage is then applied to identify accusatory or whistleblowing-style comments. Experimental results show that the combination of domain-trained Word2Vec embeddings, GMM-based clustering, and a Random Forest classifier achieves high precision and recall, even under severe class imbalance. These findings demonstrate that lightweight, domain-adapted NLP architectures can effectively support risk identification and enhance transparency in public procurement systems without requiring large-scale computational infrastructure.
Bryan Torres, Daniel Riofrío, J. Vega-Sánchez et al.· 0 citations
The rapid advancement of generative Artificial Intelligence (AI) presents challenges to academic integrity due to potential misuse like plagiarism. This study develops a text detection system specifically for the Indonesian language using a Deep Learning approach with a Bidirectional Long Short-Term Memory (Bi-LSTM) architecture. The research methodology follows the Cross-Industry Standard Process for Data Mining (CRISP-DM) framework. A dataset comprising 5,008 text rows was compiled via web scraping from journalism platforms and academic journals indexed in SINTA 4 for human-written texts, while AI-generated counterparts were engineered using ChatGPT and Google Gemini paraphrases. Text features were extracted using a Keras Tokenizer and Embedding Layer with 64 dimensions. Evaluation of the trained Bi-LSTM model on a 30% validation split demonstrated an overall accuracy of 78.24% and a Mean Absolute Error (MAE) of 0.3295. Specifically, the model achieved a 93.77% success rate in identifying human-written texts, though it logged a lower detection rate of 62.62% for academic AI text structures. The final model was successfully deployed as a web application using Streamlit.
Rafil Moehamad Alif, Syariful Alam, Chandra Dewi Lestari· Jurnal Teknologi Informatika...· 0 citations
The rapid proliferation of digitally distributed news has made large-scale automated verification an urgent research priority, as deceptive content spreads across social platforms faster than manual reviewers can evaluate it. This work introduces a two-stage deep learning pipeline in which a pretrained BERT encoder generates rich contextual token embeddings that are subsequently processed by a Bidirectional Long Short-Term Memory (BiLSTM) network, enabling the classifier to exploit both sentence-level semantics and document-level narrative flow simultaneously. Experiments on the WELFake benchmark [1] following a standardised cleaning pipeline—lowercasing, URL stripping, tokenisation, stop-word removal, and WordNet lemmatisation— yielded 98.7 % accuracy and an F1-score of 0.986. This surpasses a BERT-only baseline by 3.1 percentage points and exceeds several previously published state-of-the-art results. The gains confirm that coupling transformer-based semantic representations with recurrent sequential modelling produces a measurable and reproducible improvement in misinformation detection. Planned extensions include multilingual evaluation and knowledge-distilled encoder replacements targeting real-time throughput constraints.
Ardra P Namboodiri, Archa P S, Honey Mol O· 2026 6th International Confe...· 0 citations
The meteoric rise of large language models (LLMs) has reshaped natural language processing, inspiring new approaches to data integration tasks such as entity matching (EM). While proprietary models like GPT-4 deliver strong performance, open-source alternatives (e.g., Mistral, DeciLM, Gemma-3) require supervised fine-tuning (SFT) to excel in specialized domains. However, naïvely training billion-parameter scale LLMs on uncurated corpora is computationally prohibitive and environmentally costly, often taking days while introducing inefficiencies that limit rapid application deployment. We introduce AssistEM, a framework for efficient LLM adaptation to EM via principled data selection. AssistEM integrates NER and TF-IDF signals into a composite score, filters entity pairs with domain-level thresholds, constructs denoised semantic embeddings, and applies graph-based clustering with class-aware sampling to obtain a compact yet representative set of ∼\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sim $$\end{document}8000 high-quality training instances. This yields rapid specialization: AssistEM-trained LLMs converge in under 4.5 hours–versus 12+ hours for DeciLM-7B and 24+ hours for Yi-34B–while surpassing GPT-4-0613 by 4.3 F1 points and Ditto by 11 F1 points across diverse EM benchmarks. By aligning data quality with model specialization, AssistEM demonstrates that selective fine-tuning not only accelerates adaptation but also improves training efficiency (requiring fewer GPU hours), enabling open-source LLMs to rival–and in some cases outperform–closed-source models. These results highlight data selection as a critical lever for sustainable, domain-specific LLM deployment.
John Bosco Mugeni, S. Lynden, Toshiyuki Amagasa et al.· International Journal of Dat...· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduAug 18, 2026
A new method for surgically removing training examples from a model reveals that as datasets grow, the link between what a model learns and what it produces dissolves.
