Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducements

作者：Shu Yang, S. S. Zhu, Zeyu Wu, Keyu Wang, Junchi Yao, Junchao Wu, Lijie Hu, Mengdi Li, Derek F. Wong, Di Wang · 年份：2025 · DOI：10.18653/v1/2025.findings-acl.226 · 被引用次数：3 · 研究领域：Imbalanced Data Classification Techniques、Financial Distress and Bankruptcy Prediction、Auction Theory and Applications

We introduce Fraud-R1 , a benchmark designed to evaluate LLMs' ability to defend against internet fraud and phishing in dynamic, real-world scenarios.Fraud-R1 comprises 8,564 fraud cases sourced from phishing scams, fake job postings, social media, and news, categorized into 5 major fraud types.Unlike previous benchmarks, Fraud-R1 introduces a multiround evaluation pipeline to assess LLMs' resistance to fraud at different stages, including credibility building, urgency creation, and emotional manipulation.Furthermore, we evaluate 15 LLMs under two settings: (i) Helpful-Assistant, where the LLM provides general decision-making assistance, and (ii) Role-play, where the model assumes a specific persona, widely used in real-world agent-based interactions.Our evaluation reveals the significant challenges in defending against fraud and phishing inducement, especially in role-play settings and fake job postings.Additionally, we observe a substantial performance gap between Chinese and English, underscoring the need for improved multilingual fraud detection capabilities.The source code and dataset for this benchmark is publicly available at: