Multimodal deep learning using on-chip diffractive optics with in situ training capability

作者：Junwei Cheng, Chaoran Huang, J. W. Zhang, Bo Wu, Wenkai Zhang, Xinyu Liu, Jiahui Zhang, Yuqi Tang, Hailong Zhou, Qiming Zhang, Miṅ Gu, Jianji Dong, Xinliang Zhang · 发表于：Nature Communications · 年份：2024 · DOI：10.1038/s41467-024-50677-3 · 被引用次数：119 · 研究领域：Neural Networks and Reservoir Computing、Optical Network Technologies、Photonic and Optical Devices

Abstract Multimodal deep learning plays a pivotal role in supporting the processing and learning of diverse data types within the realm of artificial intelligence generated content (AIGC). However, most photonic neuromorphic processors for deep learning can only handle a single data modality (either vision or audio) due to the lack of abundant parameter training in optical domain. Here, we propose and demonstrate a trainable diffractive optical neural network (TDONN) chip based on on-chip diffractive optics with massive tunable elements to address these constraints. The TDONN chip includes one input layer, five hidden layers, and one output layer, and only one forward propagation is required to obtain the inference results without frequent optical-electrical conversion. The customized stochastic gradient descent algorithm and the drop-out mechanism are developed for photonic neurons to realize in situ training and fast convergence in the optical domain. The TDONN chip achieves a potential throughput of 217.6 tera-operations per second (TOPS) with high computing density (447.7 TOPS/mm 2 ), high system-level energy efficiency (7.28 TOPS/W), and low optical latency (30.2 ps). The TDONN chip has successfully implemented four-class classification in different modalities (vision, audio, and touch) and achieve 85.7% accuracy on multimodal test sets. Our work opens up a new avenue for multimodal deep learning with integrated photonic processors, providing a potential solution for low...