Detecção de mudança de distribuição em dados sequenciais

Abstract

This work aims to study the performance of the conformal method for distribution change detection (dataset shift) in sequential data, where observations are obtained sequentially over time. In particular, we consider cases where the observations are univariate, multivariate, and provided in batches (datasets) in predictive contexts. The conformal method generates p-values at each time step, which provide evidence regarding a potential change in the data distribution. Unlike classical approaches, such as the Cumulative Sum Control Chart (CUSUM), Shirayev-Roberts, and Posterior Probability methods, the conformal method does not rely on parametric assumptions about the data distribution before and after the change occurs. The conformal method was applied using different nonconformity measures proposed in this work, including those based on Kullback-Leibler divergence and various distances in Euclidean spaces constructed through statistical depth measures. The effectiveness of this approach for detecting distributional changes was evaluated in different empirical studies by measuring the average detection delay and the proportion of false alarms. It is expected that this research can contribute to the application of this method in the online monitoring of machine learning models in dynamic environments, allowing the identification of the appropriate time to recalibrate or retrain algorithms in a more reliable manner.