NSF-CSSI-Poster-OAC-2513656-Pandey

Abstract

This project develops a real-time, incremental, and sustainable sequence search index over the NIH Sequence Read Archive (SRA). The SRA is a rich but largely untapped repository containing over 50 petabytes of raw, unassembled sequencing reads, the vast majority of which is effectively inert because no scalable, affordable tool exists to search it. Building on the PI's Mantis index, the project designs novel data structures, compression algorithms, and distributed indexes that make petabyte-scale sequence search economical on commodity infrastructure, andexposes the resulting index through an easy-to-use API and a publicly hosted “Index-as-a-Service” so that scientists worldwide can query the SRA directly and accelerate biological discovery.Specifically, the project pursues six goals: (1) an advanced Mantis index that associates k-merabundance and experiment metadata for information-rich analytics; (2) an in-memory, minimizer-based meta-index that filters queries and lets the index scale efficiently to disk; (3) a dynamic and distributed Mantis index that supports incremental updates and real-time queries with communication-efficient, load-balanced partitioning across nodes; (4) an easy-to-use API in C++, Rust, and Python for versatile biological queries; (5) a publicly hosted distributed index on cost-effective commodity infrastructure; and (6) collaborations with domain scientists to enable advanced biological analyses and hypothesis testing over the SRA. The work aligns with the NSF CISE Office of Advanced Cyberinfrastructure (OAC) and the Directorate for Biological Sciences (BIO), and all software artifacts are released as open source.