Multi-Instance Multi-Task Learning for Joint Clinical Outcome and Genomic Profile Predictions From the Histopathological Images

作者：Wei Shao, Hang Shi, Jianxin Liu, Yingli Zuo, Liang Sun, Tiansong Xia, Wanyuan Chen, Peng Wan, Jianpeng Sheng, Qi Zhu, Daoqiang Zhang · 发表于：IEEE Transactions on Medical Imaging · 年份：2024 · DOI：10.1109/tmi.2024.3362852 · 被引用次数：19 · 研究领域：AI in cancer detection、Digital Imaging for Blood Diseases、Radiomics and Machine Learning in Medical Imaging

With the remarkable success of digital histopathology and the deep learning technology, many whole-slide pathological images (WSIs) based deep learning models are designed to help pathologists diagnose human cancers. Recently, rather than predicting categorical variables as in cancer diagnosis, several deep learning studies are also proposed to estimate the continuous variables such as the patients' survival or their transcriptional profile. However, most of the existing studies focus on conducting these predicting tasks separately, which overlooks the useful intrinsic correlation among them that can boost the prediction performance of each individual task. In addition, it is sill challenge to design the WSI-based deep learning models, since a WSI is with huge size but annotated with coarse label. In this study, we propose a general multi-instance multi-task learning framework (HistMIMT) for multi-purpose prediction from WSIs. Specifically, we firstly propose a novel multi-instance learning module (TMICS) considering both common and specific task information across different tasks to generate bag representation for each individual task. Then, a soft-mask based fusion module with channel attention (SFCA) is developed to leverage useful information from the related tasks to help improve the prediction performance on target task. We evaluate our method on three cancer cohorts derived from the Cancer Genome Atlas (TCGA). For each cohort, our multi-purpose prediction tasks range f...