FHS-adapter: fine-grained hierarchical semantic adapter for Chinese landscape paintings generation

作者：Xianlin Peng, Qiyao Hu, Fangqing Fan, Penglin Xie, Yihan Zhang, Rui Cao · 发表于：Heritage Science · 年份：2024 · DOI：10.1186/s40494-024-01370-7 · 被引用次数：8 · 研究领域：Generative Adversarial Networks and Image Synthesis、Computer Graphics and Visualization Techniques、Aesthetic Perception and Analysis

Abstract How to migrate text-to-image models based on pre-trained diffusion models to adapt them to domain generation tasks is a common problem. In particular, the generation task for Chinese landscape paintings with unique characteristics suffers from a scarcity of fine-grained contextual details specific to such artwork. Moreover, the use of substantial amounts of non-landscape painting data during pre-training predisposes the model to be swayed by alternative visual styles, thereby leading to generated images that inadvertently lack the distinctive traits inherent to Chinese paintings. In this paper, we propose a Fine-grained Hierarchical Semantic Adapter for Chinese landscape paintings generation, namely FHS-adapter. The method orchestrates the diffusion process in a batch-wise manner, leveraging external fine-grained multi-perspective information to guide it. It gradually diminishes the influence of other style images embedded in the pre-trained diffusion model, ultimately preserving a greater number of landscape painting elements. The encoder was also replaced with the Taiyi-CLIP encoder, which is adapted for Chinese. We propose T2ICLP, a multimodal dataset containing 10,000 high-quality image-text pairs of Chinese landscape paintings. Unlike previous datasets, this dataset extracts fine-grained textual information from four perspectives, including Meta , Description , Sentiment , Poem . We compared the proposed model with the mainstream diffusion-based T2I models. Thro...