Reconstructing Perceptive Images from Brain Activity by Shape-Semantic\n GAN

作者：Tao Fang, Yu Qi, Gang Pan · 发表于：arXiv (Cornell University) · 年份：2021 · DOI：10.48550/arxiv.2101.12083 · 被引用次数：20 · 研究领域：Cell Image Analysis Techniques、Visual Attention and Saliency Detection、Image Processing Techniques and Applications

Reconstructing seeing images from fMRI recordings is an absorbing research\narea in neuroscience and provides a potential brain-reading technology. The\nchallenge lies in that visual encoding in brain is highly complex and not fully\nrevealed. Inspired by the theory that visual features are hierarchically\nrepresented in cortex, we propose to break the complex visual signals into\nmulti-level components and decode each component separately. Specifically, we\ndecode shape and semantic representations from the lower and higher visual\ncortex respectively, and merge the shape and semantic information to images by\na generative adversarial network (Shape-Semantic GAN). This 'divide and\nconquer' strategy captures visual information more accurately. Experiments\ndemonstrate that Shape-Semantic GAN improves the reconstruction similarity and\nimage quality, and achieves the state-of-the-art image reconstruction\nperformance.\n