Descrevendo regiões de imagens através de redes neurais profundas e abstract meaning representation

Abstract

The world around us is composed of images that often need to be translated into words. This translation can take place in parts, converting regions of the image into textual descriptions. The description of the region of an image is the transformation of the information contained in this area into words in natural language, to express the way objects relate to each other. Recently, computational models that seek to perform this task in a similar way to human beings are being proposed, mainly using deep neural networks. As a way to improve the quality of the sentences produced by one of these models, this work verified the employability of the Abstract Meaning Representation (AMR) semantic representation in the generation of descriptions for image regions. AMR was investigated as representation formalism, as an alternative to natural language, using it with some variations, so that the machine learning model, using deep neural networks, was able to predict sentences in such representation. The hypothesis of this study, that the use of sentences in the form of AMR would result in better descriptions, was partially confirmed, since the model trained with AMR was superior in almost all evaluations.