On the meaning potentials of pragmatic (micro-)gestures
(2014) 2nd MaMuD – Mapping Multimodal Dialogue Workshop — Location: KULeuven, Leuven (21.November.2014)
Files
Bolly_MaMuD_PPT.pdf
Open Access - Adobe PDF
- 19.1 MB
Details
- Authors
- Abstract
- According to their traditional definition, gestures are visible bodily actions that are intentional and meaningful in a communicative context (Kendon 2004). As such, they can be attributed with the following functions (Colletta et al. 2009): (i) a reference function (deictic or representational); (ii) a discourse-structuring function (e.g., beats or cohesive devices); (iii) an expressive function (oriented towards attitudes, mental states, stance or emotions); (iv) an interactive function (oriented towards the interlocutor and the regulation of speech). In contrast to representational gestures, the hypothesis is that non-representational gestures are visible bodily actions that are idiosyncratic, (mostly) unintentional and serving pragmatic purposes in language interaction. As such, they play a role similar to that of pragmatic markers in speech (Aijmer 2013): they are metalinguistic indicators of the speaker’s mental processes and, at the same time, help the addressee to build a meaningful holistic representation of the information conveyed. A particular attention will thus be paid here to non-representational spontaneous gestures, which act as emphasizing, mitigating or punctuating devices in language communication (called adaptors, beats, batons, or motor movements – see Ekman & Friesen 1969, McNeill 1992, Krauss et al. 2000). The following research questions will be addressed: How can we decide which nonverbal units must be accounted for to reach a better understanding of pragmatic competence in human-human interaction? To what extent is it possible (or even, necessary) to integrate non-representational gestures into a consistent model for the annotation of multimodal communication? The present study is part of the CorpAGEst project (2013-2015), which aims to establish the gestural and verbal profile of very old people, looking at their pragmatic competence from a naturalistic perspective. Within this context, a multimodal corpus has been created, which is comprised of 18 semi-directed, face-to-face interviews between an adult and a very old subject (9 subjects; 16.8 hrs; approx. 250,000 words). This corpus served as a basis for the annotation of nonverbal data (hand gestures, body gestures and facial expressions). Hand gestures were decomposed into phases and annotated in terms of physical parameters (configuration, orientation, movement and position) (Bressem 2008). Body gestures were annotated taking into account the following articulators: head, shoulders, arms, trunk, legs, and feet. It is worth noting that all potential meaningful units were identified as strokes in the first step of the annotation process, including micro-movements (Ex. 1) and activities (Ex. 2). In line with the MUMIN project (Allwood et al., 2004), facial expressions were identified according to their location in the face (eyebrow, eye movement, gaze, mouth, lips) and then annotated in terms of physiological features. They were also attributed with an emotion label recognized from the face (see Bolly, to appear in 2014). Preliminary results indicate that the use of nonverbal resources is highly idiosyncratic. For instance, it appeared from a functional analysis of hand gestures that the distribution is not homogeneous among the participants. In addition, focusing on physiological patterning from face and gaze expressions in one of the speaker’s speech, no clear physiological pattern seems to be emotion-specific. Some regularity has nevertheless been noticed for the most frequent emotions used (e.g., surprise is mainly expressed by means of eyebrow raising, often combined with an exaggerated opening of the eyes). This multimodal and multi-level approach will give new insight into the use of (non)verbal pragmatic markers in relation to the participants’ emotional and attitudinal behavior in intergenerational interaction.
- Affiliations
Citations
Bolly, C. (2014). On the meaning potentials of pragmatic (micro-)gestures. 2nd MaMuD – Mapping Multimodal Dialogue Workshop, KULeuven, Leuven. https://hdl.handle.net/2078.5/190710