Aprimorando a detecção de Testes Flaky ​orientados por reexecuções com orquestração de contêineres

Abstract

Software testing is a primary activity to achieve good quality standards, where developers rely on automated tests to identify and fix software bugs and validate solutions. These tests are sometimes unreliable and present non-deterministic results, such as passing and failing when executing the same code and test several times. This type of inconsistent test is called flaky test. While the literature presents many initiatives to identify flaky tests, rerunning the tests several times remains the primary approach to debugging and analyzing test flakiness. This master’s thesis presents an approach that enhances rerun-driven flaky test detection by applying container orchestration so that flaky tests are detected faster, and more flaky tests are uncovered. To do so, we employ container orchestration technologies like Kubernetes and other tools, like Grafana, Prometheus, and Loki. The proposed approach is instantiated in a tool called FlakyTestLab. We evaluated FlakyTestLab with twenty-eight open source projects, being fourteen in Java and fourteen in Python. Using a High-End computer, FlakyTestLab performs reruns on average 76.93% faster than a baseline, and is 300% faster than the baseline in finding the first failure. FlakyTestLab detected flaky tests up to 143.40% more than the baseline. On the other hand, we observe that the flakiness ratio decreased when FlakyTestLab employed more parallelism.