Privacy-Preserving Hierarchical Reinforcement Learning Framework for Task Offloading in Low-Altitude Vehicular Fog Computing

作者：Zhiwei Wei, Jingxin Mao, Bing Li, Rongqing Zhang · 发表于：IEEE Open Journal of the Communications Society · 年份：2024 · DOI：10.1109/ojcoms.2024.3457023 · 被引用次数：22 · 研究领域：Vehicular Ad Hoc Networks (VANETs)、Transportation and Mobility Innovations、Human Mobility and Location-Based Analysis

Vehicular Fog Computing (VFC) is a promising paradigm in intelligent transportation systems (ITS), which offloads computation-intensive tasks to mobile fog nodes for real-time and low-latency services. In the forthcoming era of low-altitude economy, Unmanned Aerial Vehicles (UAVs) are being integrated as task-carrying entities into the ITS, and the novel low-altitude VFC is witnessing new challenges, introduced by dynamic UAV missions, high mobility, and privacy concerns. To preserve the offloading privacy and enhance the offloading performance in the dynamic low-altitude VFC, in this paper, we facilitate the learning-based methods and propose a hierarchical federated reinforcement learning framework. The framework consists of two levels: the local level provides Deep Reinforcement Learning (DRL) models for task vehicles and UAVs, and the cross-regional contextual level for coordinating the local experiences. At the local DRL level, we design an Attention-enhanced Federated Proximal Policy Optimization (AFedPPO) algorithm to enable decentralized training and execution (DTDE) for task offloading, which is privacy-preserving, effective, and scalable for the low-altitude VFC systems. At the cross-regional level, we introduce a contextual clustering and personalized (CCP) federated learning (FL) mechanism, which adaptively aggregates the local experiences according to the regional features. Extensive simulation results validate an average 35% improvement of the proposed framework...