Architectural redesign and evaluation of an open source MLOps platform: a case study of Apache Marvin-AI

Abstract

Machine learning is a term linked to data science, a multidisciplinary area that encom- passes knowledge of computer science, mathematics, and domain experience. Given this multidisciplinary nature, a wide variety of challenges are presented to its practitioners, as a wide range of skills is required to train models and put them into production. Part of these challenges can be solved with the help of machine learning tools and platforms. In this context, the Apache Marvin-AI is an open-source machine learning platform that offers a standardized way to develop and put machine learning models into production. While Apache Marvin-AI has a lot to offer for novices and data scientists who do not have the software engineering skills to deal with the aforementioned issues, it lacks features desired by more advanced users. To solve this problem, an architectural evolution and evaluation was carried out. The process was guided by a simplified version of ATAM (Architecture Tradeoff Analysis Method), adapted to work on a distributed open-source development en- vironment. The results of this process were analyzed in four different ways: (i) source code static analysis; (ii) feedback from stakeholders; (iii) taxonomy analysis to assess the ma- turity of the developed solutions; and (iv) an assessment of the new monitoring features. Overall, the process of designing, implementing, and evaluating the new architecture was deemed successful by all four independent evaluations, and the lessons learned are impor- tant contributions from this work.