Tiered Affordances in Web Archive Formats (TAWAF): Efficient Analytics from Indexes to Raw Captures at Scale

Abstract

Web archive collections are increasingly large and heterogeneous, making it challenging for practitioners to choose efficient representations and workflows for common analytical tasks. This half-day, hybrid tutorial provides a hands-on introduction to web archiving file formats (including WARC, CDX/CDXJ, Parquet, and derived formats such as WAT and WET) and demonstrates how each format affords different kinds of analysis. Using the End of Term (EOT) Archive Datasets (over 2.5 PB) as a real-world testbed, we present a three-tier model of analytical queries ranging from lightweight aggregate statistics over indexes and columnar files, to metadata- and text-centric analyses over derived formats, to compute-intensive studies requiring access to raw WARC payloads. Participants will learn how to access EOT data on AWS S3, convert and query CDX(J) data in Parquet, perform example analyses using SQL (DuckDB) and Python workflows, and carry out link-structure and basic NLP tasks using WAT/WET. The tutorial highlights practical trade-offs among formats (size, cost, and computational requirements) and equips attendees with concrete tools and patterns for scalable web archive collection analysis.