ReviewQA: a relational aspect-based opinion reading dataset
Quentin Grail, Julien Perez
Conférence sur l'apprentissage automatique (CAP), Rouen, France, 20 - 22 June 2018
Deep reading models for question-answering have demonstrated promising performance over the last couple of years. However current systems tend to learn how to cleverly extract a span of the source document, based on its similarity with the question, instead of seeking for the appropriate answer. Indeed, a reading machine should be able to detect relevant passages in a document regarding a question, but more importantly, it should be able to reason over the important pieces of the document in order to produce an answer when it is required. To motivate this purpose, we present ReviewQA, a question-answering dataset based on hotel reviews. The questions of this dataset are designed to require different competencies to be answered. Indeed, each question comes with an associated type that characterizes the required competency. With this framework it is possible to benchmark models and to get an overview of what are the strengths and the weaknesses of a given model on the set of tasks evaluated in this dataset. It contains more than 500.000 questions in natural language over 100.000 hotel reviews. In this setup, the answer of a question does not need to be extracted from a document, like in most of the recent datasets, but selected among a set of candidates that contains all the possible answers to the questions of the dataset. Finally, we present several baselines over this dataset.