Estudo do desempenho de LLMs como geradores automáticos de teste unitário para programas Python

Abstract

Software testing is a fundamental stage in system development, essential for detecting faults and ensuring the quality of the final product, yet it remains challenging for dynamically typed languages such as Python. This study examines the ability of the ChatGPT-3.5-turbo language model to generate test sets for Python programs, comparing its performance with tools such as Pynguin and with existing test suites. For this evaluation, 40 Python programs were used, from which Pytest-compliant tests were generated through the OpenAI API under varying temperature settings. The tests were validated using Pytest, and coverage and mutation scores were obtained with Coverage, MutPy, and Cosmic-Ray. The results show that ChatGPT-3.5-turbo produces valid tests for simple programs but maintains an overall average below 28%, despite its low computational cost. Higher temperatures improve performance, and combining tests generated at different temperatures increases diversity, allowing the model to outperform both Pynguin and existing test suites in decision coverage and mutation score.