Domain-Adaptive Pretraining Methods for Dialogue Understanding

作者：Association for Computational Linguistics 2021, Lifeng Jin, Linfeng Song, Linqi Song, Han Wu, Kun Xu, Haisong Zhang · 发表于：Open MIND · 年份：2021 · 研究领域：Psychology、Computer science、Cognitive psychology、Natural language processing、Artificial intelligence、Linguistics

Language models like BERT and SpanBERT pretrained on open-domain data have obtained impressive gains on various NLP tasks. In this paper, we probe the effectiveness of domain-adaptive pretraining objectives on downstream tasks. In particular, three objectives, including a novel objective focusing on modeling predicate-argument relations, are evaluated on two challenging dialogue understanding tasks. Experimental results demonstrate that domain-adaptive pretraining with proper objectives can significantly improve the performance of a strong baseline on these tasks, achieving the new state-of-the-art performances.