Aug 2026· Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2· 0 citations· 14 references
TL;DR
Crucially, the analysis reveals that the backbone encoder becomes self-contained post-training: the memory module facilitates the internalization of motif semantics into the model parameters, allowing the backbone to retain performance advantages even when the memory is detached during inference.
Abstract
RNA motifs and Low-Complexity Repeats (LCRs)—recurrent, conserved sequences of nucleotides—serve as the fundamental vocabulary of RNA structure and biological function. While current RNA foundation models have revolutionized sequence modeling through Transformer architectures, they predominantly prioritize modeling global dependencies within the raw sequence. To address this limitation, we propose MotRNA, a motif-aware RNA language model integrating an Explicit N-gram Memory mechanism. Unlike standard implicit neural modeling, MotRNA employs a conditional memory module to explicitly retrieve and fuse motif embeddings based on input k-mers. This mechanism provides direct access to strictly defined local patterns, complementing the global context modeled by self-attention. We validate MotRNA on large-scale RNA datasets. The model achieves a 9.6% absolute improvement in Masked Language Modeling (MLM) accuracy at a 30% masking ratio compared to state-of-the-art baselines. Notably, MotRNA maintains high robustness under extreme masking ratios, indicating effective biological signal reconstruction from sparse context. Crucially, our analysis reveals that the backbone encoder becomes self-contained post-training: the memory module facilitates the internalization of motif semantics into the model parameters, allowing the backbone to retain performance advantages even when the memory is detached during inference.
Motivation RNA language models learn representations that support structure and function prediction, but which biological concepts their hidden states encode remains unclear. Sparse autoencoders (SAEs) decompose hidden states into interpretable features, yet have not been applied to RNA language models, where byte-pair tokenization breaks the one-token-one-nucleotide correspondence that nucleotide-level attribution assumes. Results We present SPIRAL, a layer-wise SAE analysis of BiRNA-BERT. Independent SAEs at layers 0, 5, and 11 expand each 768-dimensional hidden state into 6,144 features while preserving model behaviour (explained variance above 0.99997; masked-language-model sequence recovery near 99.7%). Tokenizer-aware offset propagation aligns features to nucleotides: at layer 5, 44.3% of tested features are significantly associated with bpRNA secondary-structure classes (mean enrichment 1.61 ×), and all 1,237 eligible features with RNAcentral RNA types. Sparse profiles raise k-nearest-neighbour balanced accuracy from 0.328 to 0.359 over dense embeddings at layer 5. Availability and Implementation Source code is available at https://github.com/SadatHossain01/SPIRAL; the code, evaluation data, and trained SAE checkpoints are archived at https://doi.org/10.5281/zenodo.21891845. Contact mrahman@cse.buet.ac.bd Supplementary information Supplementary data are presented alongside the manuscript.
M. Hossain, MD. Roqunuzzaman Sojib, Md Toki Tahmid et al.· bioRxiv· 0 citations
Parnet is a multi-task foundation model trained end-to-end on 223 eCLIP-seq experiments spanning 150 RBPs to predict base-resolution RBP binding profiles directly from RNA sequence, and establishes the RBP interactome as a compact, functionally sufficient, and interpretable basis for foundation model pretraining in RNA biology.
Background RNA inverse folding designs nucleotide sequences expected to adopt a prescribed secondary structure. Search-based solvers can optimize folding-model objectives effectively, but difficult targets can require extensive sampling, and structural optimization alone does not explicitly preserve the sequence distributions or conserved motifs of natural RNA families. Methods We developed RIFT-VAE, a Transformer-based conditional variational autoencoder that receives a context-free grammar parse-tree representation of a target secondary structure and generates nucleotide labels on the corresponding tree. The framework combines progressively richer grammar rules, self-refinement learning from generated structure-sequence pairs, and cross-entropy-method optimization in the learned latent space. We evaluated RNAfold minimum-free-energy agreement on an RNAcentral-derived test set and the EteRNA100 benchmark, compared the method with four search-based solvers under matched total time budgets, and examined GC-content control and covariance-model family annotation. Results The complete pipeline achieved RNAfold-Correct/RNAfold-MCC values of 0.833/0.994 on the RNAcentral-derived test set and 0.760/0.977 on EteRNA100. Latent-space optimization accounted for the largest increase in exact structural recovery. Under a 3,600-s total budget on EteRNA100, sequences generated by RIFT-VAE improved the exact-match rate of every tested downstream search method when used as warm starts; the largest change was observed for RNAInverse (Correct, 0.297 to 0.803; MCC, 0.505 to 0.985). The pretrained model also produced sequences with measurable correct-family covariance-model hits and supported explicit GC-content conditioning. Conclusions RIFT-VAE is best interpreted as a hybrid generative-search framework: pretraining supplies a structure- and family-informed proposal distribution, whereas latent optimization concentrates evaluations in high-scoring regions. The reported structural scores are specific to RNAfold minimum-free-energy validation and do not establish biochemical function. Orthogonal folding predictors, stricter homology-controlled splits, diversity-aware evaluation, architecture-matched dot-bracket ablations, and experimental assays remain priorities for validation.
Frozen RNA-type evaluations show that RIBOSPAN learns state-of-the-art RNA representations, with a particularly clear advantage on long RNAs, and emerges as the strongest encoder-only RNA foundation model, achieving state-of-the-art performance in both full-transcript biological property prediction and zero-shot mutation-fitness modeling.
Ziyuan Wang, Bohao Tang, Fei Zhang et al.· 0 citations
Shifu, a framework of three coupled parts, scores a model on three axes (correctness, breadth across diverse RNAs, and whether its confidence can be trusted) rather than one number, and releases the dataset, code, and model backbones.
Gabriel Galvez, Quentin Vicens· bioRxiv· 0 citations
Abstract Deciphering sequence–function relationships in long non-coding RNAs (lncRNAs) remains challenging due to rapid evolutionary turnover and limited primary sequence conservation. Alignment-based approaches often fail to detect functional domains, and fixed-length k-mer models inadequately capture variable-length regulatory elements. Here, we introduce RNA-Lexis, a non-parametric statistical framework for unbiased discovery of candidate RNA sequence elements. RNA-Lexis applies segmentation based on local conditional probabilities to identify non-random sequence extensions, enabling detection of recurrent, variable-length motifs without prior biological assumptions. Conceptually analogous to language segmentation, the framework partitions continuous RNA sequences into statistically defined units (“xmotifs” and “cores”), providing an interpretable representation of sequence architecture. RNA-Lexis reconstructs the modular organization of well-characterized lncRNAs, including XIST and NORAD. In additional case studies, RNA-Lexis prioritized recurrent GC-rich elements in SNHG14 that were tested experimentally and shown to bind histones in RNA pulldown assays. RNA-Lexis also identified recurrent LINC01001 core motifs that overlap chromatin interaction patterns detected by GRID-seq. These analyses support the use of RNA-Lexis to nominate candidate sequence elements for functional follow-up, while biological function remains dependent on orthogonal experimental validation. RNA-Lexis provides a statistically grounded and interpretable framework for motif-level analysis of lncRNAs. Rather than directly inferring function, the method identifies recurrent sequence architecture and prioritizes candidate elements for mechanistic testing.
Haim Y. Bar, Amit Felach, A. Bester· Nucleic Acids Research· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
