The Importance of Fit-for-Purpose Evaluation for Computer Vision in Archaeology: A Case Study in Extracting Geospatial Data from Legacy Archaeological Maps
TL;DR
The lesson for archaeology is this: standard CV benchmarks alone cannot tell us whether an automated output is fit for archaeological inference, and Archaeology would benefit from discipline-specific fit-for-purpose metrics, like the Gini coefficient, as a regularized part of computer vision studies.
Abstract
Hand-drawn field maps are an irreplaceable archive of archaeological sites, yet converting scanned images of analog field maps into geospatial data remains a significant bottleneck. Currently, prompt-driven large language models (LLMs) cannot reproduce human heads-up digitizing consistently or affordably at scale. LLMs do, however, allow users to quickly produce custom computer vision (CV) workflows. We describe an example of one such custom Python (3.13.9) pipeline that extracts polygons that represent terraces on maps of fortified settlements (called pā) in Aotearoa New Zealand. It starts with scanned archaeological maps, decides if the map is in a cartographic style it can process, finds shapes, and decides if these shapes are valid based on a model trained on human-annotated examples. It works extremely quickly, outputs clean geospatial datasets, and has excellent spatial and formal accuracy, with a mean intersection-over-union (mIoU) for matched detections of 0.89–0.94. Detection performance, as measured by F1 scores covered an extremely wide range, from 0.800 to 0.00, with a mean of 0.528 (median = 0.588) typical of marginally successful CV studies. In practice, for every four terraces correctly mapped, three are missed, and three false positive are created. Critically, when we did a fit-for-purpose evaluation using an archaeologically meaningful metric (the Gini coefficient), we discovered a sliding scale of reliability. The best results were comparable with human-traced maps (F1 scores > 0.80). Below that, introduced error meant they were reliable in some senses but unreliable in others (F1 0.53 to 0.80). The worst were totally unreliable for quantitative analyses (F1 < 0.53). The lesson for archaeology is this: standard CV benchmarks alone cannot tell us whether an automated output is fit for archaeological inference. Archaeology would benefit from discipline-specific fit-for-purpose metrics, like the example here, as a regularized part of computer vision studies.