Natural Is the Best: Model-Agnostic Code Simplification for Pre-trained Large Language Models

作者：Yan Wang, X. Li, Tien N. Nguyen, Shaohua Wang, Chao Ni, Ling Ding · 发表于：Proceedings of the ACM on software engineering. · 年份：2024 · DOI：10.1145/3643753 · 被引用次数：6 · 研究领域：Software Engineering Research、Software Testing and Debugging Techniques、Software Reliability and Analysis Research

Pre-trained Large Language Models (LLM) have achieved remarkable successes in several domains. However, code-oriented LLMs are often heavy in computational complexity, and quadratically with the length of the input code sequence. Toward simplifying the input program of an LLM, the state-of-the-art approach has the strategies to filter the input code tokens based on the attention scores given by the LLM. The decision to simplify the input program should not rely on the attention patterns of an LLM, as these patterns are influenced by both the model architecture and the pre-training dataset. Since the model and dataset are part of the solution domain, not the problem domain where the input program belongs, the outcome may differ when the model is pre-trained on a different dataset. We propose S lim C ode , a model-agnostic code simplification solution for LLMs that depends on the nature of input code tokens. As an empirical study on the LLMs including CodeBERT, CodeT5, and GPT-4 for two main tasks: code search and summarization, we reported that 1) the removal ratio of code has a linear-like relation with the saving ratio on training time, 2) the impact of categorized tokens on code simplification can vary significantly, 3) the impact of categorized tokens on code simplification is task-specific but model-agnostic, and 4) the above findings hold for the paradigm–prompt engineering and interactive in-context learning. The empirical results showed that S lim C ode can improve the...