Self-Supervised Learning for Real-World Super-Resolution From Dual and Multiple Zoomed Observations

作者：Zhilu Zhang, Ruohao Wang, Hongzhi Zhang, Wangmeng Zuo · 发表于：IEEE Transactions on Pattern Analysis and Machine Intelligence · 年份：2024 · DOI：10.1109/tpami.2024.3379736 · 被引用次数：10 · 研究领域：Advanced Image Processing Techniques、Advanced Vision and Imaging、Photoacoustic and Ultrasonic Imaging

In this paper, we consider two challenging issues in reference-based super-resolution (RefSR) for smartphone, (i) how to choose a proper reference image, and (ii) how to learn RefSR in a self-supervised manner. Particularly, we propose a novel self-supervised learning approach for real-world RefSR from observations at dual and multiple camera zooms. Firstly, considering the popularity of multiple cameras in modern smartphones, the more zoomed (telephoto) image can be naturally leveraged as the reference to guide the super-resolution (SR) of the lesser zoomed (ultra-wide) image, which gives us a chance to learn a deep network that performs SR from the dual zoomed observations (DZSR). Secondly, for self-supervised learning of DZSR, we take the telephoto image instead of an additional high-resolution image as the supervision information, and select a center patch from it as the reference to super-resolve the corresponding ultra-wide image patch. To mitigate the effect of the misalignment between ultra-wide low-resolution (LR) patch and telephoto ground-truth (GT) image during training, we first adopt patch-based optical flow alignment to obtain the warped LR, then further design an auxiliary-LR to guide the deforming of the warped LR features. To generate visually pleasing results, we present local overlapped sliced Wasserstein loss to better represent the perceptual difference between GT and output in the feature space. During testing, DZSR can be directly deployed to super-sol...