A framework for the automated distribution and execution of tests on mobile devices

Abstract

Software testing of mobile applications is challenging due to the inherent peculiarities of mobile devices and their adaptability to diverse context execution environments. Factors such as variations in operating systems, hardware specifications, connectivity options, user interfaces, and context‐sensitive capabilities can potentially lead to deviations from the expected behaviour of an application. The different permutations of these factors gives rise to what is referred to as the Test Scenario Explosion Problem, making it unfeasible and cost‐prohibitive for mobile testing teams to test every possible scenario to which a mobile app may be subjected. Furthermore, current mobile testing approaches, including the uses of emulators, on‐premise device labs, beta testing methodologies, and cloud device farms, exhibit limitations in terms of diversity, replication of context‐sensitive scenarios, cost‐effectiveness, and representation of realistic deployment environments. In response to these challenges, our study introduces TestMate, an Android mobile testing framework designed to investigate how the Test Scenario Explosion Problem can be mitigated through the distribution and execution of automated tests on remotely connected devices. Our approach capitalises on in‐the‐field testing, leveraging the com‐ munity as a large‐scale, diverse, and authentic testing environment. Our evaluation affirms that the developed domain‐specific language, providing the ability to define the complex configuration space, and its seamless integration with the TestMate framework fulfils most of the functional requirements crucial for an effective mobile test distribution and execution framework. The system’s usability assessment yielded an above‐average SUS score of 81.88, underscoring the implementation of streamlined approach and well‐integrated solution. However, experimental results re‐ veal challenges in efficiently managing an increasing number of parallel test executions, while user study participants emphasised the risks associated with security and data privacy aspects. Overall, our study presents a promising stride toward in‐vivo testing of mobile applications. While we acknowledge the limitations of our work, addressing these constraints could contribute to the release of higher quality mobile apps.