Victoria Lin*, Jeffrey M. Girard*, Michael A. Sayette, Louis-Philippe Morency
Proceedings of the 22nd ACM International Conference on Multimodal Interaction, Pages 548–557
Publication year: 2020

Emotional expressiveness captures the extent to which a person tends to outwardly display their emotions through behavior. Due to the close relationship between emotional expressiveness and behavioral health, as well as the crucial role that it plays in social interaction, the ability to automatically predict emotional expressiveness stands to spur advances in science, medicine, and industry. In this paper, we explore three related research questions. First, how well can emotional expressiveness be predicted from visual,linguistic, and multimodal behavioral signals? Second, how important is each behavioral modality to the prediction of emotional expressiveness? Third, which behavioral signals are reliably related to emotional expressiveness? To answer these questions, we add highly reliable transcripts and human ratings of perceived emotional expressiveness to an existing video database and use this data to train, validate, and test predictive models. Our best model shows promising predictive performance on this dataset (RMSE=0.65,R2=0.45,r=0.74). Multimodal models tend to perform best overall, and models trained on the linguistic modality tend to outperform models trained on the visual modality. Finally,examination of our interpretable models’ coefficients reveals a number of visual and linguistic behavioral signals—such as facial action unit intensity, overall word count, and use of words related to social processes—that reliably predict emotional expressiveness.

2 Responses to “Toward Multimodal Modeling of Emotional Expressiveness”

  1. Jeffrey Girard

    View this project on the Open Science Framework: https://osf.io/umzar/

  2. Jeffrey Girard

    This paper was nominated for the Best Paper Award at ICMI 2020, congratulations Victoria!

Leave a Reply to Jeffrey Girard Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.