Introduction to the special issue on multimodal corpora for modeling human multimodal behavior

Abstract

There is an increasing interest in multimodal communication as suggested by several national and international projects (ISLE, HUMAINE, SIMILAR, CHIL, AMI, CALO, VACE, CALLAS), the attention devoted to the topic by well-known institutions and organizations (the National Institute of Standards and Technology, the Linguistic Data Consortium), and the success of conferences related to multimodal communication (ICMI, IVA, Gesture, Measuring Behavior, Nordic Symposium on Multimodal Communication, LREC Workshops on Multimodal Corpora). As Dutoit et al. (2006) lament, however, « there is a lack of multimodal corpora suitable for the evaluation of recognition/synthesis approaches and interaction strategies … one must admit that most corpora available today target the study of a limited number of modalities, if not one ». Corpora are not only relevant to evaluation purposes, their importance extending to all the stages of design and development of multimodal systems. Moreover, established practices and guidelines are also missing concerning their design, the number and types of levels they should contain, etc. Indeed, multimodal corpora are expensive to collect and to annotate: they require multiple levels of annotations (various properties of speech and gestures, facial expressions, location of people, body posture, etc.) possibly at different levels of abstractions (actions and events, relational behavior). Despite these problems, however, several interesting attempts at developing multimodal corpora have been recently conducted, and the important insights and experience gained in these projects deserve systematization and dissemination. It seems, therefore, that the time is ripe to offer an overview of these efforts in this special issue. Our focus is on multimodal corpora and their use for representing and modeling human behavior. We have also included communication studies that contribute to the definition of collection protocols, coding schemes, and reliable models of multimodal human behavior that can be built from corpora and assessed against previous non-digital experimental approaches from the social sciences. Out of 28 submitted papers, 11 were selected for publication. They illustrate the need of a wide-angle approach for tackling multimodal research corpora: theory and applied aspects, formal and informal annotations, large corpus versus small corpus, manual versus automatic annotation are only some of the issues involved. The papers cover three main topics: the first five provide insights on multimodal communication and its phenomena (emotion, irony, explanation, feedback, sequence and turn-taking). The following four papers describe corpus-based approaches to embodied conversational agents (facial expressions, gestures and their combinations). Finally, the last two papers deal with an area of multimodal interpersonal communication which has been recently receiving much attention: meetings. For each of these three topics, we provide below a short overview of the state of the art as well as short summaries of the papers published in this special issue.