Construção de um pipeline de dados utilizando serviços da nuvem

Abstract

The evolution of the computer network and the increasing access and interaction of the world population to the internet has provided a change in the data scenario. At all times, data is generated in exorbitant amounts and of the most varied structures, breaking with conventional data systems that were focused on transactional operations, initiating a process where systems have evolved to meet the analytical demands that grow with the concept of process orientation and decisions through information (Data Driven). In the era of Big Data, in addition to the evolution of data models and infrastructure for processing and storage, there was also the specialization of professionals in the area so that each one had mastery over specific processes of the data life cycle. Following this context, the objective of the present study is to build an understanding of the Big Data scenario and its influence on the evolution of current processes and concepts in the area, carrying out a practical development of the creation of a data pipeline solution using computing services in cloud to integrate, collect, model, process and analyze COVID-19 data and world development indicators.