The attribute selection for GRE challenge : overview and evaluation results

Abstract

The Attribute Selection for Generating Referring Expressions (ASGRE) Challenge was the first shared-task evaluation challenge in the field of Natural Language Generation. Six teams submitted a total of 22 systems. All submitted systems were tested automatically for minimality, uniqueness and ‘humanlikeness’. In addition, the output of 15 systems was tested in a task-based experiment where subjects were asked to identify referents, and the speed and accuracy of identification was measured. This report describes the ASGRE task and the five evaluation methods, gives brief overviews of the participating systems, and presents the evaluation results.