Explorando a avaliação de sumários automáticos multidocumento multilíngues

Abstract

Multilingual Multi-Document Automatic Summarization (MMDS) is a computational task through which a summary is produced in a target language from a collection of at least two news stories which address the same subject, one in the user’s language and the other(s) in foreign language(s). The scientific literature shows that not many researches approach methods which generate summaries in Portuguese. Based on the CF and CFUL summarization methods, the present thesis describes the development of a study whose goal was to refine the summary quality evaluation, by varying (i) the native language of the producers of the reference summaries, that is, summaries written by human subjects after reading the corresponding source texts and which are necessary for the automatic calculation of informativeness, and (ii) the compression rate (desired summary size). Furthermore, this thesis outlines the enlargement of the corpus used for the investigation of these methods through the addition of texts in German (the original corpus included content in Portuguese and English) and the production of four extracts for each of the twenty clusters. The results show that the reference summaries are slightly impacted by their writer’s native language, even though additional factors might be taken into account, such as the size of each source text and the content compatibility. Regarding the summarization methods, this study found that extracts with a lower compression rate performed better when it came to the automatic evaluation of informativeness and worse in the assessment of linguistic quality.