A Psychological Strategy Annotation Method Using Multiple LLMs with a Chain of Thought Based on Deductive Reasoning

作者：Jinran Wang, Jiaming Luo, So‐Young Yang, Yongjie Zhou, Xuefang Zhang, Rongfeng Su, Nan Yan, Lan Wang · 年份：2025 · DOI：10.1109/apsipaasc65261.2025.11249074 · 研究领域：Mental Health via Writing、Topic Modeling、Machine Learning in Healthcare

Using appropriate psychological counseling strategies is critical for improving the quality of large language modeldriven mental health dialogues, yet a single large language model (LLM) often produces unreliable results due to ambiguous strategy definitions and degradation of thought in reasoning. Multi-LLM voting-based strategy annotation methods are often hampered in their overall decision-making performance due to the low accuracy of some individual models. To address this, we propose a multi-LLM system with a deductive-reasoning chain of thought. We built a dataset that comprises psychologist-client dialogue segments, each annotated with nine strategies by four professional psychologists. The system uses three LLMs as experts to generate initial strategy candidates, and then an LLM as a judge applies deductive reasoning-aligning dialogue evidence with strategy definitions-to annotate the optimal strategy. Experiments show that this approach outperforms single models, voting, and conventional chain-of-thought methods in F1 score and accuracy. This work advances LLM-based psychological strategy annotation, offering a reliable tool for mental health dialogue systems.