TSGET: Two-Stage Global Enhanced Transformer for Automatic Radiology Report Generation

作者：Xiulong Yi, You Fu, Ruiqing Liu, Hao Zhang, Rong Hua · 发表于：IEEE Journal of Biomedical and Health Informatics · 年份：2024 · DOI：10.1109/jbhi.2024.3350077 · 被引用次数：55 · 研究领域：Topic Modeling、Multimodal Machine Learning Applications、Natural Language Processing Techniques

Recently, automatic radiology report generation, which targets to generate multiple sentences that can accurately describe medical observations for given X-ray images, has gained increasing attention. Existing methods commonly employ the attention mechanism for accurate word generation. However, such attention-based methods fail to leverage useful image-level global features, thereby limiting the model's reasoning ability. To tackle this challenge, we propose two-stage global enhancement layers to facilitate the Transformer to generate more reliable reports from a global perspective. Specifically, the 1st Global Enhancement Layer (1st GEL) is designed to capture the global visual context features by establishing the relationships between image-level global features and previously generated words. The 2nd Global Enhancement Layer (2nd GEL) is devised to capture the region-global level features by building the relationships between image-level global features and region-level information. The experiments demonstrate that by integrating the aforementioned two-stage global enhancement layers into the Transformer model, our proposal achieves state-of-the-art (SOTA) performance on various Natural Language Generation (NLG) evaluation metrics. Further Clinical Efficacy (CE) evaluations also validate that our proposal is able to predict more critical information.