Diretrizes para a Documentação de Engenharia de Dados em uma abordagem DevDocOps

Abstract

The dissertation presents an in-depth investigation into the integration of agile software development practices, such as Continuous Integration (CI) and Continuous Deployment (CD), with the DevOps philosophy, which seeks collaboration between development and operations teams. The research explores the context of Data Engineering, which deals with the development, implementation, and maintenance of systems and processes for transforming raw data into high-quality information. The study highlights the importance of technical documentation of software and data in an agile development environment and how automating this process can improve the quality and efficiency of software deliveries. The DevDocOps approach is introduced as a way to integrate continuous documentation into the software de velopment lifecycle, addressing the challenges of balancing adequate documentation with effective communication. Furthermore, the research investigates the automation of documentation for data models developed in SAP HANA, a powerful in-memory Database Management System (DBMS). Documentation of Calculation Views, a key object in SAP HANA, is explored in detail, emphasizing the importance of accurate documentation for understanding and effective use of these models. The study also compares various data documentation automation tools available in the market and discusses their limitations in relation to the DevDocOps approach. It concludes that while these tools offer efficient solutions for data management and documentation, they often do not fully meet the needs of continuous documentation integration into the development lifecycle. To prove the practical application of the proposed approach, an experimental study was conducted where the findings from the literature were applied in a real development environment. In this study, the DevDocOps approach was implemented to integrate continuous documentation into the data engineering development lifecycle, with a focus on automating the SAP HANA data model documentation process. Finally, the dissertation emphasizes the importance of a holistic approach to technical documentation in agile development environments, especially in Data Engineering, and proposes guidelines for the successful implementation of the DevDocOps approach. The study contributes to advancing knowledge on the integration of agile practices, DevOps, and technical documentation in complex software development and data engineering environments.