Research on Multimodal Emotion Recognition Based on Fusion of Electroencephalogram and Electrooculography

作者：Jialai Yin, Minchao Wu, Yan Yang, Ping Li, Fan Li, Wen Liang, Zhao Lv · 发表于：IEEE Transactions on Instrumentation and Measurement · 年份：2024 · DOI：10.1109/tim.2024.3370813 · 被引用次数：20 · 研究领域：Emotion and Mood Recognition、Gaze Tracking and Assistive Technology、EEG and Brain-Computer Interfaces

Emotion recognition plays a vital role in building a harmonious society and emotional interaction. Recent research has demonstrated that multi-modal inter-channel correlations and insufficient emotion elicitation plague deep learning-based emotion identification techniques. To cope with these problems, we propose a transformer model of multi-modal and channel attention fusion (MCAF-Transformer). First, we employ an olfactory video approach to evoke emotional expression more fully and acquire EEG and EOG signal data. Second, the model makes full use of multi-modal channel information, time-domain and spatial-domain information of EEG and EOG signals, captures the correlation of different channels using channel attention, and improves the accuracy of emotion recognition by focusing on the global dependence on the temporal order using the transformer. We conducted extensive experiments on the olfactory video sentiment dataset, and the experimental results were correct at 94.63%. The results show that olfactory videos evoke emotion more adequately than pure videos and that the MCAF-Transformer model significantly outperforms other emotion recognition methods.