FlakeFighters: A Pytest Plugin to Detect Flaky Test Failures

Abstract

Flaky tests — tests which non-deterministically pass and fail against the same version of code — pose a well-established painful issue for developers due to the unpredictability of failures. The current state of practice to detect and work around flaky tests is to simply rerun them. However, this can erode confidence in failing test outcomes and represents a significant waste of time and compute resources. A better solution is to detect flaky tests based on static data (collectable without running any tests) and/or execution data (collected by running the test suite) to determine if a test failure has occurred due to flaky behaviour or a genuine bug. Several such techniques have appeared in recent literature, but these exist in isolation as prototype research tools, and typically only support Java codebases. This paper presents FlakeFighters, an extensible pytest extension that implements a "Swiss army knife" of flaky test classification tools to help developers classify the underlying cause of failing tests as genuine bugs or flakiness in Python codebases. We ran pytest with the FlakeFighters extension on 23 failing workflows from the home-assistant/core repository and found that it was able to identify all of the failing flaky tests. A video demonstration of FlakeFighters can be found at https://youtu.be/3J4odtgzQX0.