Improving the effectiveness of incremental mutation testing

Abstract

With software systems becoming more complex, the need to have good quality code is becoming increasingly important. Confidence in the code can only be gained if one has confidence in the test suite which checks that code. Mutation testing is a technique which can be used to find shortcomings in a test suite. It is a fault injection testing technique, which creates faulty versions of a program, and checks if the test suite catches the faults. Although effective, it is also computationally expensive, making it unattractive to be used in software development environments. As an improvement, incremental mutation testing was introduced. This is a technique which exploits the iterative nature of agile environments, and performs mutation testing on only the changed parts of the code, greatly reducing the execution time. Although the execution time decreases considerably, the parts of the code that depend on the changed parts are not included in incremental mutation testing. This in turn results in a problem, because faults can surface when the parts which are dependent on the changed parts are executed. Since the dependent parts are not included in incremental mutation testing, it is not possible to check whether these new faults are caught by the test suite. This work addresses this problem by introducing a tool which improves the effectiveness of incremental mutation testing. By using static analysis on the source code and generating a call graph, the parts of the code that depend on those changed are also included as part of the incremental mutation testing process. This made incremental mutation testing more effective, by detecting shortcomings in the test suite which can appear on the tests that execute the dependent parts of the changed methods. The results obtained in the evaluation have shown us that this tool is effective on codebases which have test suites that require a long time to execute. On the other hand, it was also shown that it was not feasible to use the tool on codebases which have test suites that execute in a short amount of time.