SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting

作者：Jianrong Xu, Jiayi Pan, Yongkang Zhou, S. Chen, Jinhao Li, Yaoxiu Lian, Junyi Wu, Guohao Dai · 年份：2025 · DOI：10.1145/3695053.3730996 · 被引用次数：6 · 研究领域：Topic Modeling、Natural Language Processing Techniques、Data Quality and Management

Early exiting has recently emerged as a promising technique for accelerating large language models (LLMs) by effectively reducing the hardware computation and memory access.In this paper, we identify that the LLM vocabulary serves as the runtime search space of the early exiting predictor and significantly influences the predictor workload (e.g., 20% overall inference latency with 3 10 4 vocabulary size in Llama2).We propose a novel paradigm using speculative models to reduce this search space, while addressing three critical challenges for further predictor optimization.(1) Time-consuming predictor with high computational complexity.Current predictor designs leverage basic models with high-dimensional input that ignore inherent data variation and GPU parallelization opportunities, resulting in 15% overall inference latency.(2) Under-utilization of layer-wise predictor deployment.Current early exiting systems treat the predictor in each layer equally without considering the activation frequencies of layer-wise predictors, leading to 20% inference overhead.(3) Exponential mapping complexity of predictor in speculative decoding.Each token in the token tree of speculative decoding is treated as an independent search space when applying the current early exiting mapping, leading to exponential mapping complexity and failing to incorporate the high-throughput benefits