Simulating Conversational Search Users with Parameterized Behavior

作者：Ivan Sekulić, Lili Lu, Navdeep Singh Bedi, Fábio Crestani · 年份：2024 · DOI：10.1145/3673791.3698425 · 被引用次数：7 · 研究领域：Topic Modeling、Advanced Text Analysis Techniques、Speech and dialogue systems

User simulation is emerging as a promising direction towards scalable and reliable training and evaluation of conversational search systems. As such, the simulated user assumes the user's role in interaction with the system and aims to satisfy its information needs by querying, answering clarifying questions, and providing feedback. While recent research made significant progress in generating user's utterances, it remained limited to simulating the average user. In other words, state-of-the-art simulators do not take into account differences that exist between real users, such as personality and behavioral traits. To this end, we propose a framework for incorporating behavioral traits into a generative user simulator for conversational search. Specifically, we utilize in-context learning to embed behavioral traits, such as cooperativeness and politeness, into the LLM-based simulator. The framework, dubbed ParamConvSim, parametrizes certain behavioral traits and allows tuning the degree of a single trait present in the simulated user. In this paper, we design a user model, modeling the user's patience, cooperativeness, and politeness. In addition, we present and analyze the effectiveness of conversational search systems when interacting with different such simulated users. Results suggest that different simulated users indeed interact differently with the search system, leading to different system effectiveness levels.