Avaliação de geradores automáticos de dados de teste com ênfase no teste de mutação

Abstract

Context: With the increasing presence of technology in daily life, it is crucial to guarantee the quality of software products, which can be reached by testing activity. However, testing is highly expensive, once it is responsible for, in most cases, more than half of the software development cost. Among the existing testing criteria, mutation testing is highlighted once it is capable of revealing specific faults in software. Although mutation testing is effective, test data generation and live mutants analysis make the activity unfeasible because it is still required considerable human intervention. Goals: Therefore, this paper aims at evaluating four automatic test data generators (EvoSuite, JTeXpert, Palus, and Randoop), focusing on mutation testing, by measuring adequacy, effectiveness and cost. The first three of these generators implement heuristics during test data generation and they are evaluated both individually and combined, using as baseline a randomic generator. Method: Such generators are compared by conducting an experiment with 33 software products implemented in Java. Adequacy is measured regarding statement coverage, effectiveness regarding mutation score and cost regarding the number of generated test cases. For each generator we generated 10 different test sets for each one of the 33 analysed programs, totaling 1320 distincts test sets. Results: Results show, when we analyze the generators individually, it is not possible to conclude there is a difference between them. However, when we combine the heuristic-based generators, it is possible to conclude there is significant improvement when compared to the randomic generator. Moreover, when we combine all generators, including the randomic one, we obtained improvments of 0,2% in statement coverage and 4% in effectiveness, but we also obtained a significant increase in cost. Conclusion: In general, we observe there is a complementary aspect between the three automatic test data generators and they should be combined to improve the quality of test sets regarding mutation score. However, even with such combination, there is still 25% of the generated mutants which remain alive, indicating there is still room for improvements on quality of the used generators.