Massive Scale Data Analytics at LCLS-II

作者：Jana Thayer, Zhantao Chen, Richard O. Claus, D. S. Damiani, C. J. B. Ford, M. M. Dubrovin, Victor Elmir, W. Kroeger, Xiang Li, Stefano Marchesini, Valerio Mariani, Riccardo Melcchiori, S. Nelson, Ariana Peck, Amedeo Perazzo, Frédéric Poitevin, Christopher O’Grady, Julieth Otero, Omar Quijano, Murali Shankar, Monarin Uervirojnangkoorn, Riccardo Veraldi, Matthew Weaver, Clemens Weninger, Seshu Yamajala, Cong Wang, Chun Hong Yoon · 发表于：EPJ Web of Conferences · 年份：2024 · DOI：10.1051/epjconf/202429513002 · 被引用次数：9 · 研究领域：Particle Accelerators and Free-Electron Lasers、Distributed and Parallel Computing Systems、Particle Detector Development and Performance

The increasing volumes of data produced at light sources such as the Linac Coherent Light Source (LCLS) enable the direct observation of materials and molecular assemblies at the length and timescales of molecular and atomic motion. This exponential increase in the scale and speed of data production is prohibitive to traditional analysis workflows that rely on scientists tuning parameters during live experiments to adapt data collection and analysis. User facilities will increasingly rely on the automated delivery of actionable information in real time for rapid experiment adaptation which presents a considerable challenge for data acquisition, data processing, data management, and workflow orchestration. In addition, the desire from researchers to accelerate science requires rapid analysis, dynamic integration of experiment and theory, the ability to visualize results in near real-time, and the introduction of ML and AI techniques. We present the LCLS-II Data System architecture which is designed to address these challenges via an adaptable data reduction pipeline (DRP) to reduce data volume on-thefly, online monitoring analysis software for real-time data visualization and experiment feedback, and the ability to scale to computing needs by utilizing local and remote compute resources, such as the ASCR Leadership Class Facilities, to enable quasi-real-time data analysis in minutes. We discuss the overall challenges facing LCLS, our ongoing work to develop a system responsive...