GAppium experiment: Applying gamification to mobile GUI testing to increase effectiveness, efficiency, engagement

Abstract

Testing of mobile applications is a crucial activity that aims to reveal bugs and defects affecting software running on our smartphones. Usually, mobile applications feature complex Graphical User Interfaces (GUIs) as the only interface with the human user, therefore proper End-to-End testing of mobile applications should also include GUI testing. In recent years, several automated testing methods have been proposed for mobile GUI testing; however, due to the dynamic and interactive nature of GUIs, they often fail to create realistic use cases. Exploratory testing is a popular GUI testing technique that involves testers freely exploring the application under test. This technique heavily relies upon testers’ experience and creativity, and is negatively influenced by human factors like stress and boredom, which are likely to appear in unmotivated testers. Gamification, i.e., the introduction of game design mechanics to tasks of other nature, has recently gained much popularity in business contexts, being applied to a wide range of software-related activities. In this article, we propose a gamification approach to mobile Exploratory Testing to improve the effectiveness, efficiency, and engagement of the software testers. We measure the impact of gamification with an in-vivo experiment involving students from a master’s degree software engineering course, adopting a fully-randomized experiment design. On the efficiency side, results show a positive impact of gamification on the explored pages – a measure of effectiveness – and on the explored pages per unit of time – which we adopted as an efficiency measure. Moreover, a final survey presented to students reveals that most of them believe that it is a good idea to use gamification in Exploratory Testing, and that they would use gamification in future if they had to do Exploratory Testing. Finally, the majority of participants report having fun while using the system. Our results highlight that gamification features can have a significant impact if introduced in a classical testing environment, possibly increasing effectiveness, efficiency and perceived engagement of the testers. We believe this research could serve as a starting point for researchers approaching the mobile testing environment, as well as industry practitioners working in the field.