[Editorial] Proceedings of LREC2020 Workshop "People in language, vision and the mind" (ONION2020)

Abstract

The ability to adequately model and describe people in terms of their body and face is interesting for a variety of language technology applications, e.g., conversational agents and interactive narrative generation, as well as forensic applications in which people need to be identified or their images generated from textual or spoken descriptions. Such systems need resources and models where images associated with human bodies and faces are coupled with linguistic descriptions. Thus, the research needed to develop such datasets and models is placed at the interface between vision and language research, a cross-disciplinary area which has received considerable attention in recent years, e.g., through the activities of the European Network on Integrating Vision and Language (iV&L Net), the 2015–2018 Language and Vision Workshops, the 2018–2019 Workshops on Shortcomings in Vision and Language and the ongoing Multi-Task, Multilingual, Multimodal (Multi3Generation) Generation COST Action. The aim of this first edition of the ONION workshop was to provide a forum to present and discuss current research focusing on multimodal resources as well as computational and cognitive models aiming to describe people in terms of their bodies and faces, including their affective state as it is reflected physically. Such models might either generate textual descriptions of people, generate images corresponding to descriptions of people, or in general exploit multimodal representations for different purposes and applications. Knowledge of the way human bodies and faces are perceived, understood and described by humans is key to the creation of such resources and models, therefore the workshop also invited contributions where the human body and face are studied from a cognitive, neurocognitive or multimodal communication perspective. Recent research on the analysis of images and text or the generation of image descriptions focused on datasets which might contain people as a subset; however, we argue that such general multimodal resources are not adequate for the specific challenges posed by applications based on the modelling of human bodies and faces. Descriptions of people are frequent in human communication, for example when one seeks to identify an individual or distinguish one person from another, or in the course of conveying a person’s affective state on the basis of facial expression, posture etc. These descriptions are also pervasive in descriptive or narrative text. Depending on the context, they may focus on physical attributes, or incorporate inferred characteristics and emotional elements. Human body postures and faces are being studied by researchers from different research communities, including those working with vision and language modeling, natural language generation, cognitive science, cognitive psychology, multimodal communication and embodied conversational agents. The workshop aimed to reach out to all these communities to explore the many different aspects of research on the human body and face, including the resources that such research needs, and to foster cross-disciplinary synergy.