Optimized Scheduling of Spark Workflows in Multi-Cloud Environments With Deadline and Budget Constraints

作者：Kamran Yaseen Rajput, Xiaoping Li, Abdullah Lakhan, Abdul Rasheed Mahesar, Dileep Kumar Sajnani · 发表于：IEEE Transactions on Cloud Computing · 年份：2025 · DOI：10.1109/tcc.2025.3628548 · 被引用次数：2 · 研究领域：Cloud Computing and Resource Management、Distributed and Parallel Computing Systems、Big Data and Digital Economy

To overcome vendor lock-in and reliability issues in single-cloud deployments, organizations increasingly adopt multi-cloud environments. However, scheduling Spark workflows across heterogeneous clouds under simultaneous deadline and budget constraints remains challenging due to resource diversity, variable pricing, and cross-cloud data transfers. We propose the Deadline Budget Spark Workflow Scheduling to Multi-Cloud (DB-SWSMC) algorithm, a novel scheduling algorithm combining heuristic initialization with simulated annealing optimization to: (1) efficiently allocate resources while balancing cost-time tradeoffs, (2) handle intra/inter-cloud data dependencies, and (3) rigorously enforce constraints. Evaluations across five workflows and compared against existing algorithms (HBDCWS, DBCS, and BDHEFT). Experimental results demonstrate that DB-SWSMC outperforms existing algorithms by 20-40% in cost efficiency and 15-80% in success rates, especially under tight budget and deadline constraints.