Laemple: A Benchmarking Framework for Virus Lineage Deconvolution Tools for SARS-CoV-2 from Wastewater
TL;DR
A modular, expandable, simulation-based benchmarking framework as a reproducible Snakemake workflow, named Laemple, to evaluate the performance of virus lineage deconvolution tools, revealing substantial variation in tool performance across conditions.
Abstract
Background. Correct and accurate deconvolution of SARS-CoV-2 lineages from wastewater sequencing data is a challenging task, given the intricacy of wastewater amplicon-sequencing data and the ever-growing complexity of the lineage classification. Existing benchmarking studies made use of artificial spike-in compositions, thereby falling short of reflecting the prevailing complexity of wastewater samples. Results. We present a modular, expandable, simulation-based benchmarking framework as a reproducible Snakemake workflow, named Laemple, to evaluate the performance of virus lineage deconvolution tools. Using in silico simulated data sets of varying complexity and sequencing quality, we demonstrate its utility by evaluating seven publicly available tools, based on their precision, sensitivity, and reproducibility. Freyja showed robust sensitivity and consistent performance across diverse data set complexities, alongside user-friendly installation and documentation, while VaQuERo demonstrated the highest precision. Conclusions. Our results reveal substantial variation in tool performance across conditions, emphasizing the need to benchmark with diverse and complex scenarios. This framework enables informed tool selection for researchers and public health agencies and allows developers to stress-test their tools during development and maintenance, i.e., updating the lineage definition for newly emerging virus lineages. To this end, Laemple was designed in a modular fashion for customization and future expansion to support ongoing software development across all stages of the application life-cycle management.
