MCFHNet: Multi-Channel Fusion Hybrid Network for Efficient EEG-fNIRS Multi-modal Motor Imagery Decoding

作者：Jiaming Chen, Dan Wang, Bo Hu, Weibo Yi, Meng Xu, Dingrui Chen, Qing Zhao · 发表于：2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) · 年份：2022 · DOI：10.1109/embc48229.2022.9871385 · 被引用次数：9 · 研究领域：EEG and Brain-Computer Interfaces、Neuroscience and Neural Engineering、Advanced Memory and Neural Computing

Motor Imagery-based Brain Computer Interface (MI-BCI) is a typical active BCI with a main focus on motor intention identification. Hybrid motor imagery (MI) decoding methods that based on multi-modal fusion of Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), especially deep learning-based methods, become popular in recent MI-BCI studies. However, the fusion strategy and network design in deep learning-based methods are complex. To solve this problem, we proposed the multi-channel fusion method (MCF) to simplify current fusion methods, and we designed a multi-channel fusion hybrid network (MCFHNet) based on MCF. MCFHNet combines depthwise convolutional layers, channel attention mechanism, and Bidirectional Long Short Term Memory (Bi-LSTM) layers, which enables strong capability of feature extraction in spatial and temporal domain. The comparison between MCFHNet and representative deep learning-based methods was performed on an open EEG-fNIRS dataset. We found the proposed method can yield superior performance (mean accuracy of 99.641 % in 5-fold cross validation of an intra-subject experiment). This work provides a new option for multi-modal MI decoding, which can be applied in the rehabilitation field based on hybrid BCI systems.