Log Anomaly Detection via Transformers Pre-Trained on Massive Unlabeled Data

作者：Senming Yan, Lei Shi, Jing Ren, Wei Wang, Limin Sun, Wei Zhang · 发表于：IEEE Transactions on Network Science and Engineering · 年份：2026 · DOI：10.1109/tnse.2026.3654089 · 被引用次数：1 · 研究领域：Computer Science

Cyber attacks pose serious threats to computer systems. Automatically detecting anomalous patterns in system logs is critical for identifying and mitigating security risks. However, as log data grows increasingly complex and labeled logs remain scarce, existing detection methods face significant challenges. To address these issues, we introduce the pre-training and fine-tuning paradigm for log analysis and propose a hybrid pipeline tailored for accurate and low-cost log anomaly detection. Specifically, we employ a masked log reconstruction strategy to pre-train a Transformer encoder–based foundation model by leveraging the sequential dependencies in unlabeled logs. The model is then fine-tuned on an event prediction task to derive the anomaly detector. To reduce computational and storage overhead, we further design a knowledge distillation method tailored for compressing log anomaly detectors. Beyond fitting the detector's outputs, our method also exploits its internal representations to transfer richer knowledge. Experiments on the HDFS, BGL, and Thunderbird public datasets demonstrate that our framework outperforms state-of-the-art baselines in multiple metrics. Empirical evaluation on a reconstructed HDFS dataset confirms that it can adapt to real-world scenarios where labeled data is scarce. Moreover, through our knowledge distillation approach, the lightweight detectors achieve outstanding performance with substantially lower overhead, while maintaining robustness in rea...