Drift-Aware Agentic ETL for Schema-Evolving Data Lakes

作者：W Wang, Ye Zhang, Zichen Pan, Ziruo Ke, Yilin Sun, Yang Yun · 年份：2026 · DOI：10.1109/gaiis69281.2026.11519158 · 被引用次数：15 · 研究领域：Data Stream Mining Techniques、Advanced Database Systems and Queries、Time Series Analysis and Forecasting

Schema evolution in data lakes frequently causes downstream ETL pipeline failures, silent data corruption, and metric drift. Existing approaches address schema matching or data cleaning in isolation but lack a closed-loop mechanism that detects structural changes, generates repair strategies, and validates corrections before deployment. We propose Drift-Aware Agentic ETL (DAETL), a framework that integrates schema drift detection with LLM-powered repair agents constrained by data contracts, historical lineage, and sample-based validation. Central to our design is a Repair Confidence Score (RCS) that combines semantic matching confidence, historical transformation consistency, sample verification pass rate, and downstream distribution divergence to decide between automatic patch acceptance and human escalation. We evaluate DAETL on a controlled schema evolution benchmark spanning relational, semi-structured JSON, and event-stream datasets with five categories of injected drift. Results show that DAETL achieves 92.4% pipeline recovery rate and 91.8% data correctness under controlled drift injections, outperforming rule-based, detection-only, and unconstrained LLM baselines. Ablation studies confirm that each constraint source contributes measurably to repair quality.