Cluster-Based I/O Analysis of Scientific Workflows

作者：Peng Jj, Jiaqiang Ma, Yixin Zhang, Weizhao Lin, Lihua Yang · 年份：2025 · DOI：10.1109/aiahpc66801.2025.11290245 · 研究领域：Scientific Computing and Data Management、Distributed and Parallel Computing Systems、Advanced Data Storage Technologies

Scientific workflow serves as a critical framework supporting large-scale scientific discovery. However, their I/O performance has become the bottleneck in Exascale computing environment. Existing analysis methods either examine the I/O behavior of jobs in isolation, or rely on expert to manually interpret complex data dependencies, making it difficult to efficiently uncover I/O patterns in workflows. To address this, we propose a cluster-based automated I/O analysis approach. This approach constructs two types of clustering objects, intra-job and inter-job, and designs multi-dimensional I/O features to characterize them. The Clustering results are mapped onto the workflow structure for intuitive I/O pattern identification. We conduct a case study on real scientific workflow, demonstrating that our method can effectively identify job clusters with similar I/O behaviors. In particular, we can automatically detect data transfer patterns such as aggregation and broadcast in workflow, which pinpoint the I/O bottlenecks. This reduces the workload of domain experts and automates the analysis of I/O patterns.