Using Causal Reasoning to Test Scientific Software

Abstract

From predicting the weather to simulating disease transmission, scientific software plays an increasingly pivotal role in developing scientific understanding that informs our everyday lives. However, they are also some of the most difficult software systems to properly test. Because the developers are often the scientists themselves, they may not be familiar with established testing techniques and are likely to focus on scientific results rather than software quality. From a technical standpoint, scientific software systems tend to exhibit several characteristics that make traditional testing techniques impractical and often insufficient. They have large, complex input spaces, are computationally expensive to run, often rely on stochastic black-box components, and are applied in exploratory contexts where the expected outcomes are not known. Causality-driven testing has recently emerged as a potential solution to these challenges. Building on the concept of metamorphic testing — investigating whether a change to the input configuration yields the expected corresponding change in output — causal testing employs domain knowledge and existing runtime data to efficiently use the limited test runs that can be performed. In this walkthrough, we will use real examples to demonstrate the limitations of traditional unit testing. We will then show how metamorphic testing can go beyond these limitations, how causal reasoning can be employed to increase the efficiency of the testing process, and how the Causal Testing Framework can automate much of this. The slides can be viewed by opening `slides.html` in your browser or `slides.pdf` in any PDF viewer. If you want to view the interactive jupyter notebook, uncompress the source zip file (or clone the repository) and follow the instructions in the README file.