Bifrost: Extending RoCE for Long Distance Inter-DC Links

作者：Peiwen Yu, Feiyang Xue, Chen Tian, Xiaoliang Wang, Yanqing Chen, Tom Wu, Lei Han, Zifa Han, Bingquan Wang, Xiangyu Gong, Wanchun Dou, Guihai Chen · 年份：2023 · DOI：10.1109/icnp59255.2023.10355634 · 被引用次数：11 · 研究领域：Software-Defined Networks and 5G、Network Traffic and Congestion Control、Network Time Synchronization Technologies

RDMA over Converged Ethernet (RoCEv2) has been widely deployed to data centers (DCs) for its better compatibility with Ethernet/IP than Infiniband (IB). As cross-DC applications emerge, they also demand high throughput, low latency, and lossless network for cross-DC data transmission. However, RoCEv2's underlying lossless mechanism Priority-based Flow Control (PFC) cannot fit into the long-haul transmission scenario and degrades the performance of RoCEv2. PFC is myopic and only considers queue length to pause upstream senders, which leads to large queueing delay. This paper proposes Bifrost, a downstream-driven lossless flow control that supports long distance cross-DC data transmission. Bifrost uses virtual incoming packets, which indicates the upper bound of in-flight packets, together with buffered packets to control the flow rate. It minimizes the buffer space requirement to one-hop bandwidth delay product (BDP) and achieves low one-way latency. Real-world experiments are conducted with prototype switches and 80 kilometers cables. Evaluations demonstrate that compared to PFC, Bifrost reduces average/tail flow completion time (FCT) of inter-DC flows by up to 22.5%/42.0%, respectively. Bifrost is compatible with existing infrastructure and can support distance of thousands of kilometers.