LLM4VV: Exploring LLM-as-a-Judge for Validation and Verification Testsuites

作者：Zachariah Sollenberger, Jay Patel, Christian Munley, Aaron Jarmusch, Sunita Chandrasekaran · 年份：2024 · DOI：10.1109/scw63240.2024.00238 · 被引用次数：5 · 研究领域：Artificial Intelligence in Law

Large Language Models (LLM) continue to improve and are revolutionizing the landscape of software development. These large models have demonstrated potential to generate, debug, test, analyze, document, and even translate code. Thus they are a valuable tool in the software development cycle. If used correctly, such tools can often accelerate the development cycle. Though the tools are powerful and new, the community is cautious of training using biased or sensitive data, which can lead to biased, dangerous, or incorrect outputs along with the inadvertent release of confidential information. Additionally, the carbon footprints and the un-explainability of these "black box" models continue to raise questions about the reliability of LLMs.With these opportunities and these challenges ahead, this paper explores the idea of "judging" LLM-generated code to better understand and "open up" the un-explainable "black box" models used by LLMs. We probe into the black box of one such LLM that has generated the best compiler tests for the directive-based programming models OpenMP and OpenACC in our earlier research. We challenge DeepSeek’s deepseek-coder-33B-instruct model with intentionally-erroneous code, and we also define relevant metrics and adopt an agent-based approach to evaluate the LLM and assess its capabilities as an LLM-as-a-judge. We also develop a pipeline-based approach to streamline the entire workflow. Finally, we make use of all of these strategies together to develop a...