Small Language Models: Survey, Measurements, and Insights

作者：Zhichun Lu, Xiang Li, Cai, Dongqi, Runduan Yi, Fangming Liu, Xiwen Zhang, Nicholas D. Lane, Mengwei Xu · 发表于：arXiv (Cornell University) · 年份：2024 · DOI：10.48550/arxiv.2409.15790 · 被引用次数：16 · 研究领域：Computational and Text Analysis Methods、Expert finding and Q&A systems、Topic Modeling

Small language models (SLMs), despite their widespread adoption in modern smart devices, have received significantly less academic attention compared to their large language model (LLM) counterparts, which are predominantly deployed in data centers and cloud environments. While researchers continue to improve the capabilities of LLMs in the pursuit of artificial general intelligence, SLM research aims to make machine intelligence more accessible, affordable, and efficient for everyday tasks. Focusing on transformer-based, decoder-only language models with 100M-5B parameters, we survey 70 state-of-the-art open-source SLMs, analyzing their technical innovations across three axes: architectures, training datasets, and training algorithms. In addition, we evaluate their capabilities in various domains, including commonsense reasoning, mathematics, in-context learning, and long context. To gain further insight into their on-device runtime costs, we benchmark their inference latency and memory footprints. Through in-depth analysis of our benchmarking data, we offer valuable insights to advance research in this field.