Open Access Repository Bern University of Teacher Education

Facial and Bodily Expressions of Emotional Engagement: How Dynamic Measures Reflect the Use of Game Elements and Subjective Experience of Emotions and Effort

Greipl, Simon; Bernecker, Katharina; Ninaus, Manuel (2021). Facial and Bodily Expressions of Emotional Engagement: How Dynamic Measures Reflect the Use of Game Elements and Subjective Experience of Emotions and Effort. Proceedings of the ACM Human-Computer Interaction, 5. 10.1145/3474667

[img] Text
3474667.pdf - Published Version
Available under License Creative Commons: Attribution-Noncommercial-No Derivative Works (CC-BY-NC-ND).

Download (1MB)

Users' emotional engagement in a task is important for performance and motivation. Non-intrusive, computerized process measures of engagement have the potential to provide fine-grained access to underlying affective states and processes. Thus, the current work brings together subjective measures (questionnaires) and objective process measures (facial expressions and head movements) of emotions to examine users' emotional engagement with respect to the absence or presence of game-elements. In particular, we randomly assigned 156 adult participants to either a spatial working memory task with or without game elements present, while their faces and head movements were recorded with a webcam during task execution. Positive and negative emotions were assessed before the task and twice during task execution using conventional questionnaires. We additionally examined whether perceived subjective effort, assumed to inherit a substantial affective component, manifests at a bodily expressive level alongside positive and negative emotions. Importantly, we explored the relationship between subjective and objective measures of emotions across the two tasks versions. We found a series of action units and head movements associated with the subjective experience of emotions as well as to subjective effort. Impacted by game elements, these associations often fit intuitively or lined up with findings from literature. As did a linear increase of blink (action unit 45) intensity relate to participants performing the task without game elements, presumably indicating disengagement in the more tedious task variant. On other occasions, associations between subjective and objective measures seemed indiscriminative or even contraindicated. Additionally, facial and bodily reactions and the resulting subjective-objective correspondences were rather consistent within, but not between the two task versions. Our work therefore both gains detailed access to automated emotion recognition and promotes its feasibility within research of game elements while highlighting the individuality and context dependency of emotional expressions.

Item Type:

Journal Article (Original Article)

PHBern Contributor:

Bernecker, K.




Jessica Brunner

Date Deposited:

12 Feb 2024 10:57

Last Modified:

13 Feb 2024 20:07

Publisher DOI:





Actions (login required)

Edit item Edit item