Um estudo sobre testes Flaky em projetos Python

Abstract

Understanding the behavior of automated tests is crucial for improving software development practices. This study focuses on investigating the incidence of flaky tests in Python projects. The research involved analyzing a diverse set of Python projects to address the proposed research questions. The results indicated that 19.44% of the analyzed projects exhibited flaky tests, a significantly higher rate than the 4.5% reported by Gruber et al. (2021). The analysis showed that the majority (85.1%) of the identified flaky tests depended on the order of execution, emphasizing the need to ensure independence between tests to minimize the chance of flakiness. Non-order-dependent \textit{flaky} tests were less frequent but exhibited inconsistencies mainly due to dependencies on external APIs and execution time limits of libraries. These findings suggest that test stability can be compromised by external factors, such as the availability of external services and the variability in responses from these APIs.

Future work could be carried out to expand the FlaPy tool to enable the installation and verification of a broader range of projects.