Rethinking Data Selection at Scale: Random Selection is Almost All You Need

作者：Tingyu Xia, Bowen Yu, Kai Dang, Yang An, Yuan Wu, Yuan Tian, Yi Chang, Junyang Lin · 年份：2025 · DOI：10.18653/v1/2025.findings-emnlp.146 · 被引用次数：2 · 研究领域：Machine Learning and Data Classification、Psychometric Methodologies and Testing、Statistics Education and Methodologies

Supervised fine-tuning (SFT) is crucial for aligning Large Language Models (LLMs) with human instructions.The primary goal during SFT is to select a small yet representative subset of training data from the larger pool, such that fine-tuning with this subset achieves results comparable to or even exceeding those obtained using the entire dataset.However, most existing data selection techniques are designed for small-scale data pools, which fail to meet the demands of real-world SFT scenarios.In this paper, we replicated several self-scoring methods-those that do not rely on external model assistance-on two millionscale datasets, and found that nearly all methods struggled to significantly outperform random selection when dealing with such largescale data pools.Moreover, our comparisons suggest that, during SFT, diversity in data selection is more critical than simply focusing on high-quality data.We also analyzed the limitations of several current approaches, explaining why they perform poorly on largescale datasets and why they are unsuitable for such contexts.Finally, we found that filtering data by token length offers a stable and efficient method for improving results.This approach, particularly when training on longtext data, proves highly beneficial for relatively weaker base models, such as Llama3.The code is available at https://github.com/ xiatingyu/SFT-DataSelection-at-scale.