Enhancing Differential Testing with LLMs for Testing Deep Learning Libraries

作者：Meiziniu Li, Dongze Li, Jianmeng Liu, Jialun Cao, Yongqiang Tian, Shing-Chi Cheung · 发表于：ACM Transactions on Software Engineering and Methodology · 年份：2025 · DOI：10.1145/3735637 · 被引用次数：6 · 研究领域：Advanced Neural Network Applications、Adversarial Robustness in Machine Learning、Software Testing and Debugging Techniques

Differential testing offers a promising strategy to alleviate the test oracle problem by comparing the test results between alternative implementations. However, existing differential testing techniques for deep learning (DL) libraries are limited by the key challenges of finding alternative implementations (called \(counterparts\) ) for a given API and subsequently generating diverse test inputs. To address the two challenges, this article introduces DLLens , a large language model (LLM)-enhanced differential testing technique for DL libraries. The first challenge is addressed by an observation that DL libraries are commonly designed to support the computation of a similar set of DL algorithms. Therefore, the counterpart of a given API’s computation could be successfully synthesized through certain composition and adaptation of the APIs from another DL library. DLLens incorporates a novel counterpart synthesis workflow, leveraging a LLM to search for valid counterparts for differential testing. To address the second challenge, DLLens incorporates a static analysis technique that extracts the path constraints from the implementations of a given API and its counterpart to guide diverse test input generation. The extraction is facilitated by LLM’s knowledge of the concerned DL library and its upstream libraries. DLLens incorporates validation mechanisms to manage the LLM’s hallucinations in counterpart synthesis and path constraint extraction. We evaluate DLLens on two popular ...