J. Mozziconacci

DNA Language Models: An Assessment of Pre-Training for Fine-Tuning Tasks

This work investigates three key questions: do transformer-based models provide sufficient improvements on fine-tuning tasks upon heavy pretraining, what is the actual contribution of pretraining in this setting, and how does BPE tokenization impact performance on genomics-related tasks.