近日，实验室博士生马绘关于多模态对话情感识别的研究成果被多媒体领域顶级期刊IEEE Transactions on Multimedia (TMM)录用。TMM属于中科院一区、CCF推荐B类以及清华计算机学科推荐A类期刊。

题目：A Transformer-based Model with Self-distillation for Multimodal Emotion Recognition in Conversations

摘要：Emotion recognition in conversations (ERC), the task of recognizing the emotion of each utterance in a conversation, is crucial for building empathetic machines. Existing studies focus mainly on capturing context- and speaker-sensitive dependencies on the textual modality but ignore the significance of multimodal information. Different from emotion recognition in textual conversations, capturing intra- and inter-modal interactions between utterances, learning weights between different modalities, and enhancing modal representations play important roles in multimodal ERC. In this paper, we propose a transformer-based model with self-distillation (SDT) for the task. The transformer-based model captures intra- and inter-modal interactions by utilizing intra- and inter-modal transformers, and learns weights between modalities dynamically by designing a hierarchical gated fusion strategy. Furthermore, to learn more expressive modal representations, we treat soft labels of the proposed model as extra training supervision. Specifically, we introduce self-distillation to transfer knowledge of hard and soft labels from the proposed model to each modality. Experiments on IEMOCAP and MELD datasets demonstrate that SDT outperforms previous state-of-the-art baselines.

  对话情感识别(Emotion recognition in conversations, ERC)，即识别对话中每个话语的情感，对于构造情感聊天机器人至关重要。现有的研究主要关注建模文本形式的上下文和说话者信息，但忽略了多模态信息的重要性。与文本模态的ERC不同，捕获话语间模态内和模态间的交互，学习不同模态的重要程度以及增强模态表示在多模态ERC中起重要作用。本文针对该任务提出一个基于Transformer和自蒸馏的模型。基于Transformer的模型利用模态内和模态间transformers捕捉模态内和模态间的交互，并设计分层门控融合策略动态地学习模态之间的权重。此外，为了学习更具表现力的模态表示，将所提出模型的软标签视为额外的训练监督。具体而言，引入自蒸馏训练，将模型硬标签和软标签包含的知识转移到每个模态中。在IEMOCAP和MELD两个数据集上的实验表明，SDT取得了比基线模型更好的性能。