A domain specific language integrating testing and runtime verification

Abstract

Quality assurance plays an important role in the ICT industry. The software being developed must be tested to ensure the implemented system obeys the business requirements provided by clients. Moreover, the final product given to the client should contain no obvious bugs. Currently, in the industry the most common technique for software verification is software testing. While this technique is good to uncover bugs, it cannot be used to verify all possible inputs. Through the addition of runtime verification, the system is verified during execution time and thus, the input values being checked are based on the execution trace and path taken by the user. Therefore, through the combination of both software testing and runtime verification it is possible to achieve a confidently tested system. However, the combination may result in the same specification being defined twice, once for each technique, thus indicating a duplication of effort. In this work, we propose a framework, Q-Code, allowing users to write specifications for system behaviour checking before deployment and during execution time, thus combining the two techniques. Through this framework users are able to write verification properties in a script, and when run, the framework will integrate the properties with the system. The proposed technique extends the tools Code Contracts and IntelliTest, by offering an approach which allows the specification of temporal properties, and the definition of manual test cases to support those generated automatically. Moreover, this framework aims to provide the industry with a simple and convenient approach of defining properties, by reducing the time spent on software testing through the use of automated testing tools, and by improving the confidence of the system through the use of runtime monitoring. The quality of Q-Code was evaluated by discussing the language in relation to the properties which define a well-designed language, such as simplicity, understandability and readability, among others. Additionally, through the use of a case study, it was possible to evaluate the expressivity of the language. The evaluations showed that the language possesses good qualities and has good expressivity; however, it may be improved to reduce limitations. Furthermore, Q-Code was also evaluated through feedback obtained from IT professionals. During the evaluation, participants were asked to understand and write properties using the Q-Code syntax and answer a set of questions related to the language and the concept. The feedback obtained was positive, showing that participants would support the use of such a framework. However, aside from the positive feedback, suggested improvements were provided.